Re: Reiser4 terribly slow
I agree, reiser4 has been very slow for me, and as I've upgraded the kernel, it's gotten, if anything, worse. I'm currently running 2.6.18-mm3. I upgraded (as I have before) in the hope that the generally very slow issue had been fixed in this kernel release, and if I knew that it was fixed in the latest mm, I would upgrade in a heartbeat. I'd like to see this fixed, as it's the only real issue that I have. Is this something that can be fixed in a reasonable time, or am I SOL and it's time to switch back to ext3? Have money problems/Han's arrest stopped development? I would think with a 1Ghz K7, I would have a fast enough processor But I don't. Read/write maxes processor usage. I have a regularly scheduled backup that rsyncs with a ext3 disk on the same machine. It slows the machine to a standstill, processor time is 90%+ system. And that's just reading. Forget burning a CD/DVD in a reasonable time. Although now that I have 1Gig of memory inseat of the 128Mb that I had before, I can CACHE the CD image and write it out in a reasonable timeframe. But accessing the disk is SLOW. I disabled fsync in the kernel, and that helped some, but not that much. I've had Reiser4 as my root file system for a number of years now, but I've just about given up. I lust after the advanced features, encryption and change logging in particular, but without a reasonably fast filesystem, I don't see the point. On Wednesday 24 January 2007 17:57, Xu CanHao wrote: AFAIK, vim fsyncs, azureus fsyncs, and may be many other applications fsyncs but not only databases. Definitly turn off fsync() is a bad idea. I just wonder how bad r4's fsync() performance is. It seems the result is: disabled fsync() r4 is even slower than enabled fsync() ext3.o Is it never be possible to improve that? I found in the mailing-list that Hans talked it for many years. this must be a CRITICAL performance flaw for r4. 2007/1/25, [EMAIL PROTECTED] [EMAIL PROTECTED]: On Wed, 24 Jan 2007 22:05:53 +0800, Xu CanHao said: extremely low performance, i managed to turn off fsync() in fs/reiser4/plugin/file/file.c (nullify the sync_unix_file() function), (Maybe you understand the following, if so, feel free to ignore. I'm mostly making sure the list archives have this note so anybody else tempted to do this will think twice)... Note that this can be *very* dangerous to the health of your database if you implement it blindly without understanding the *full* implications. Basically, applications almost never call fsync() unless they need it for database consistency. A system crash at an inopportune time *will* totally ruin your database.
Encryption/Compression and meta data (like file names)
I seem to recall several people saying that the compression/encryption plugins will only encrypt items, and not the filesystems metadata. And then saying that this might be useless for their purposes, because being able to see the filename might be enough for the attacker. However, I don't think that is correct. The only thing visible to an attacker would be the tree structure, not the item, which contains the directory information including the filenames. So at most an attacker would be able to browse the tree structure, and see the keys used and the size of the item pointed to. If this is true, selection of a good hash should entirely prevent the giving information away problem, leaving only useless key-size-location data for an attacker. Is my understanding correct?
Re: / is no longer Reiser4 :(
On Monday 21 November 2005 23:17, Hans Reiser wrote: Alexander Zarochentsev wrote: one bug responsible for fs corruption was fixed recently. the fix is in 2.6.14-mm2 already. Then send an email titled something like Data corruption bug was fixed, be sure to upgrade! to our list. I tried it again and got the complete oops text. It's a soft lockup detected on CPU#0 message, which leads me to believe that it's a side effect of the sync taking a long time. I've got 1.5 gigs of memory and a very slow hard disk. hdparm -tT gives ~4.5 MB/s or up to 8 MB/s if I have everythings turned on that I can. I can't enable dma, because hdparm refuses to do so, and I haven't figured out which parameters to pass to which modules to make it so that I can. It's also possible that my hardware is buggy, and the driver knows that and is thus refusing to enable dma and corrupt data. I've got 2.6.14-mm2 with the latest reiser4 patch, but it's giving my loads of garbage like: *** Warning: plugin_set_compression [fs/reiser4/plugin/compress/compress_plugins.ko] undefined! I think that maybe the source was corrupted in the restore process (I'll have to do it again---later) I'm moving on friday/saturday, and I don't have arrangements for internet access at the new digs yet, so if you've got questions, ask them now... BUG: soft lockup detected on CPU#0 Pid: 4582, comm: pdflush EIP: 0060:[f892da3b] CPU: 0 EIP is at ide_pio_sector+0xcb/0x120 [ide_core] EFLAGS: 0282Not tainted (2.6.14-mm1) EAX: ec5bc000 EBX: eb531000 ECX: EDX: 01f0 ESI: 0004 EDO: f893f120 ENP: 0282 DS: 007b ES: 007b CR0: 8005003b CR2: 0812b008 CR3: 298b8000 CR4: 06d0 [f892dadd] ide_pio_multi+0x4d/0x70 [ide_core] [f892de61] task_out_intr+0x101/0x140 [ide_core] [f892836d] ide_intr+0x7d/0x180 [ide_core] [f892dd60] task_out_intr+0x0/0x140 [ide_core] [c013c22d] handle_IRQ_event+0x3d/0x70 [c013c2c3] __do_IRQ+0x63/0xc0 [c0105379] do_IRQ+0x19/0x30 [c0103b1a] common_interupt+0x1a/0x20 [c013d6ca] unlock_page+0xa/0x30 [f8a4c130] write_jnodes_to_disk_extent+0x1b0/0x2c0 [reiser4] [f8a4c4c9] write_jnode_list+0xa9/0x110 [reiser4] [f8a51483] write_fq+0x53/0x70 [reiser4] [f9a47d19] write_prepped_nodes+0x39/0x40 [resier4] [f8a48f0c] squeeze_right_twig+0x10c/0x160 [reiser4] [f8a49156] squeeze_right_twig_and_advance_coord+0x26/0x80 [reiser4] [f8a49a84] handle_pos_end_of_twig+0xd4/0x290 [reiser4] [f8a810d5] item_length_by_coord+0x15/0x20 [reiser4] [f8a49f28] squalloc+0x28/0x60 [reiser4] [f8a4829f] jnode_flush+0x2cf/0x340 [reiser4] [f8a48519] flush_current_atom+0xf9/0x250 [reiser4] [f8a457cf] flush_some_atom+0xaf/0x2c0 [reiser4] [f8a565a4] writeout+0x124/0x200 [reiser4] [f8a52c04] reiser4_sync_inodes+0x64/0xf0 [reiser4] [c0184b0d] writeback_inodes+0x4d/0xb0 [c0144118] background_writeout+0x98/0xe0 [c0144cb0] pdflush+0x0/0x30 [c0144c0d] __pdflush+0xbd/0x160 [c0144cd6] pdflush+0x26/0x30 [c0144080] background_writeout+0x0/0xe0 [c0144080] background_writeout+0x0/0xe0 [c012f1e6] kthread+0xb6/0xc0 [c012f130] kthread+0x0/0xc0 [c0101369] kernel_thread_helper+0x5/0xc
/ is no longer Reiser4 :(
Following Han's comment about the deliterious effects of 6% fragmentation, I attempted a manual defrag of my hard disk. While restoring the .tar file, I had nothing better to do than watch it. And a good thing too! It got a recurring oops. about every other minute or so, it would stop with a long kernel message than mostly scrolled off of the screen... I thought those where supposed to show up in a log files somewhere if possible, but I can't find it. And it should have been possible, as the computer continued to run just fine. These oopses caused some sort of data corruption - root wouldn't boot properly afterwards. So I reformated as ext3 and untarred my root again. That worked fine, so I know it wasn't corruption of the tar file. I took a photograph, and I'll try to type in some of it. Just looking at the names of the procudures, it looks like memory pressure made reiser4 flush, and then some of the lower level functions tried to allocate memory and failed. But since I don't have the top of the oops message, I can't tell. Wait - I could've stopped the scrolling with ^S, scrolled back with ^pageup, and photoed the whole thing! Aaaargghh Well, I'm not redoing it right now, I need to be getting to bed. I may try it again later - but then maybe I'll update to 2.6.14-mm2 with patch from namesys first... Here's the (tail end of the) oops message, sans addresses and offsets because I'm feeling lazy and I'm in a hurry: mempool_alloc+0x3a/0xe0 __split_bio+0x128/0x190 in_drive_list dm_request generic_make_request submit_bio do_IRQ reiser4_clear_page_dirty write_jnodes_to_disk_extent write_jnode_list write_fq flush_current_atom flush_some_atom writeout reiser4_sync_inodes writeback_inodes background_writeout pdflush __pdflush pdflush background_writeout kthread kthread kernel_thread_helper
Re: More Slowdown
On Thursday 17 November 2005 12:40, Vladimir V. Saveliev wrote: Please try whether the attached patch improves anything. It simplifies fsync by avoid commiting of transactions which do not modify file being fsync-ed. The patch applied to 2.6.14-mm2 with warnings, but that can be ignored. I haven't tried this patch yet, but I did try the earlier 'disable fsync completely' patch. In fact, I'm using it right now. It doesn't help. Therefore, your patch probably won't help either. OK, disabling fsync does help a *little* bit. But I think that the issue (for me, anyway) isn't sync per se, it's just flat-out access time. Deleteing lots of small files is EXTREMELY slow, but even reading files is slower than it should be. It took no less that 10 minutes to delete an old kernel source tree, for instance. It's related to fragmentation, I think. I didn't really notice any speed problems until my hard drive got to about 95% (or so) full. But they haven't gone away, even though usage is now down around 54%. Hrm... I should be able to check that... About an hour later... So maybe I was wrong. 6.5% data fragmentation doesn't seem that high. But 19.8% tree fragmentation does seem a bit high. How should this data be interpreted? #measurefs.reiser4 -T /dev/mapper/e-h -f measurefs.reiser4 1.0.5 Copyright (C) 2001, 2002, 2003, 2004 by Hans Reiser, licensing governed by reiser4progs/COPYING. Tree fragmentation ... done 0.197747 #measurefs.reiser4 -S -D /dev/mapper/e-h -f measurefs.reiser4 1.0.5 Copyright (C) 2001, 2002, 2003, 2004 by Hans Reiser, licensing governed by reiser4progs/COPYING. Data fragmentation ... done 0.065593 Tree statistics ... done Packing statistics: Formatted nodes:3721.29b (90.85%) Branch nodes: 1734.57b (42.35%) Twig nodes: 2886.97b (70.48%) Leaf nodes: 3814.82b (93.14%) Node statistics: Total nodes:8543553 Formatted nodes: 146354 Unformatted nodes: 8397199 Branch nodes: 292 Twig nodes: 10542 Leaf nodes: 8532719 Item statistics: Total items: 637352 Nodeptr items: 146353 Statdata items: 191218 Direntry items: 17512 Tail items: 245018 Extent items: 36869 #
Re: More Slowdown
That return 1; should be return 0;
a return value of 1 indicates failure.
Also, maybe this should be applied also to the
asmlinkage long sys_fdatasync(unsigned int fd)
function as well? There isn't much to choose from between these two
functions...
On Wednesday 16 November 2005 12:18, Nikita Danilov wrote:
Francesco Biscani writes:
On Wednesday 16 November 2005 01:45, David Masover wrote:
I got sick of waiting for it and nuked the fsync call. All my kernels
have a custom patch such that sys_fsync just returns true, no matter
what.
Mhh.. would it be something like this?
--- buffer.c.old2005-11-16 02:36:46.129829994 +0100
+++ buffer.c2005-11-16 02:37:11.125079752 +0100
@@ -376,7 +376,7 @@
asmlinkage long sys_fsync(unsigned int fd)
{
- return do_fsync(fd, 0);
+ return 1;
}
What are the implications of doing something like this? Is sync going
to
One implication is following:
1 application like fetchmail downloads a mail message from the server
2 saves message in the mailbox
3 fsyncs the mailbox (which is a no-op in our case)
4 sends notification to the server, which deletes message
5 crash occurs (transaction made on the step 2 is not yet committed to
the disk)
6 after reboot mailbox is restored to the state it had before step 2
7 message is lost.
stop working or isn't it using this function?
Thanks,
Francesco
Nikita.
Re: Slowdown is gone apt-get works with updated reiser4. So nevermind...
On Saturday 12 November 2005 06:38, Hans Reiser wrote: Being seamless, cleanly implemented, and requiring little or no admin work, matters a lot to end users. Amen, Brother! Yes, users can do what you said with rsync, but it is important that it be no more work than specifying a --use-versioning mount option, and even that is beyond most users (but that is where defaults come in to help them). The namespace for the past versions should be as cleanly done as WAFL does them. Whether space gets freed automatically when space gets 10% is another mount option. Where we might do better than WAFL is in allowing touching filename//checkin to cause a version to get recorded, rather than doing it at particular times. Hans Of particular concern is that the name space should (somehow) allow me to easily grab version by date, even if the file hadn't changed for the two weeks before that, and in fact still hasn't changed... Make it really easy to grab all or some files by wildcard and with a specific revision, even when not every file changed with that revision. Oh, BTW. The slowdown as I called it is still there. I guess I spoke to soon. The specific symptom is that the effected process locks for a time, usually just a second or two, but sometimes a minute or two and and at least once for many many minutes. I think that the crash (soft lockup) that I reported earlier is related as well. And it sounds like the comment that rvalles had about lockups with mmaped files, except that it doesn't lock up permanently. Just for a second or three usually.
Crash with 2.6.14-mm1 (and slow...)
It seems that when my HD approached 90% full, access (write) time went up dramatically, with some programs seeming to stop entirely. I especially notice it when starting vim. The problem hasn't gone away after I got the use percentage back down around 85%. It's getting really annoying. And then I got this while trying to apt-get install cvs This is actually the second time it's locked while doing this, but the previous time I was in X and didn't see the bug message. I captured this with a digital camera and typed it back in, so there may be minor errors, though I doubt it. I haven't tried apt-get install on anything else yet. Debian GNU/Linux testing/unstable herb tty2 herb login: root Password: BUG: soft lockup detected on CPU#0! Pid: 3492, comm: apt-get EIP: 0060:[f8a6129c] CPU: 0 EIP is at writepages_unix_file+0x11c/0x280 [reiser4] EFLAGS: 0202 not tainted (2.6.14-mm1) EAX: f6188b60 EBX: ECX: 003f EDX: ESI: 03fd EDI: EBP: f646c9ec DS: 007b ES: 007b CR0: 8005003b CR2: bf9c3000 CR3: 35d79000 CR4: 06d0 [c013cfeb] __filemap_fdatawrite_range+0x6b/0x80\ [c013d030] filemap_fdatawrite+0x30/0x40 [c015114f] msync_interval+0x6f/0xe0 [c015132f] sys_msync+0x16f/0x182 [c01030d5] syscall_call+0x7/0xb _
Slowdown is gone apt-get works with updated reiser4. So nevermind...
I grabbed the reiser4 patch for 2.6.14-rc5 and compiled it. Thanks to Vladimir V. Saveliev's comments about EXPORT_SYMBOL(clear_page_dirty_for_io); in mm/page-writeback.c, I was able to get it working as a module, and it seems to have taken care of it. I would have waited for the official 2.6.14-mm1 reiser patch before upgrading to 2.6.14, but CD burning didn't work with 2.6.12. rant There is, btw, one main reason that I've decided that whatever trouble it may cause, and whatever growing pangs I may experience along the way, root reiser4 is worth it. Does anybody remember GoBack? It was a versioning system for windows 95/98 that was incredibly flexible and useful. Tracked all changes to the whole disk. Old versions of a file? no problem. grab an old version of a directory for referance temporarily? easy. Got a virus? revert the whole HD, and then grab the newer copies of your documents and saved games as needed. Microsoft includes an almost useless version of the same ability with their system restore facility on XP, but I've never seen or heard of anybody using it. And rightfully so, it majorly stinks. It doesn't track all files, it's interface is opaque, the fact that it even exists is hidden seven layers deep, you can't control which files are restored, you can't list previous versions of a file, you can't copy an old version of a subdirectory and it's contents out without wiping the new version. You can bet that in 10 years or so, Microsoft will come out with a version of system restore that doesn't suck though. Integrated into the file manager, right click access, and everything else too. Goback is the only thing that I missed when I switched over to linux, and reiser4 is the only thing I've found that even hints at a similar ability. Even if it takes another 10 years to reach the same point of usability that GoBack had, it'll be well worth it. And when that day comes, I won't even have to reformat (you didn't have to reformat to install GoBack, either.) It's been 10 years or so since my last format (Hrmm... a little over eight, actually) and I figured that as long as my HD was trashed (another reason to love reiser4 - any fs that has a standard tool that commonly trashes file systems beyond any hope of recovery... darn fsck.ext3) I might as well prepare for the future, and get better performance while I'm at it. Note though, that features are definitely the first thing for me, performance is nice but not something that I'll notice too much, and I'd definitely be willing to sacrifice some to get enhanced semantics or versioning. Compiles take forever no matter what you do, and as long as the little things (like starting vim) don't take longer than a second or two, that's good enough. /rant
Re: Our introduction to Reiser-list
On Thursday 27 October 2005 12:05, Alexander G. M. Smith wrote: John Gilmore wrote on Wed, 26 Oct 2005 17:02:06 +: I had understood that a big part of the issue with file-as-directory was the same as the issue with hard links to directories. Which I thought is that if you move one directory into another, you can lose the connection to the root of the filesystem -- I.E. ../../.. etc is no longer guaranteed to get you to / -- And of course this also means that there is no way to get from / to your freshly moved files, and you've just lost a chunk of disk space to complete inaccessibility. fsck would have to be made smarter specifically to be able to figure that one out, too. The file move operation has to check that it doesn't break the graph into two graphs, thus disconnecting some files from the root. Or you can think of it as being a way of deleting a whole subgraph of files all at once, kind of like an rmdir that works better than usual :-) Speaking of connecting to the root, one concept I found useful was to have a True Path to the root. One of the hard links to a file / directory is considered to be the main one and the rest are auxiliary (easier done if each file/dir stores a list of parents). The main one is guaranteed to lead to the root (a recursive property) and is used for .. in directories and the equivalent operation for files. Then when you want a canonical path, you build it by following the main parents up to the root. The True Path comes in useful for faking hard links for POSIX compatibility by misrepresenting them as symbolic links using the dynamically generated true path string. As well, if you remove a hard link and it wasn't the main parent then you don't have to do any graph traversal to fix up things; all items will still have a valid link to the root through their unchanged main parent. I'm getting confused. So forgive my if I become incoherent in my attempts to understand... rambling I'm failing to see the difference between a true path and a not guaranteed true path -- Don't all paths (that point upwards) have to lead to root eventually? Hrm... Maybe you mean that an upward path (also called a back reference no?) is the one that is guaranteed to not lead into a cycle? Or at least to lead back out... So this True Path is a short name for shortest path to root? Or maybe you're storing only one parent pointer, and it's the True Path - but thats very expensive to update when the true reference is unlinked. Or maybe you are talking about a backreference that isn't updated -- for efficiency reasons? Leading to stale backreferences, which of course would be more efficient only on delete, and then you'd still have to... Nevermind, no gains here. I don't see a way to (safely) move (multiple hard-linked) directories around without requiring that the new connection to root be atomically verified. And that requires parent pointers. And multiple parent pointers, which in turn makes verifying the connection to root complex, and a True Path concept might be usefull. But it might better be called a 'guaranteed path'. But it's then just one way of maintaining some information to help you decide which of several parents you're going to ascend through when verifying a new connection to root. Maybe there a better way to safely move (and delete, etc) multiple hard-linked directories? Maybe some more complex method - some rule to insure the graph stays acyclic? But there's nothing inherently wrong with cycles, they are just hard to deal with. /Rambling Wouldn't requiring parent pointers (and multiple parent pointers, at that) require that updates be done in (at least) two places every time that you did any linking/unlinking/link-changing? And wouldn't that kill performance pretty badly?
Re: Our introduction to Reiser-list
On Wednesday 26 October 2005 22:40, Nate Diller wrote: File-as-Directory The issue with file-as-directory (FaD) is that it removes the Acyclic property of the namespace graph. This is because anything which contains file data can be hard-linked, even if that item is also a directory. It would be unreasonable to discard hard links entirely, they are quite useful and would be much more useful, in fact, with FaD enabled. So the new namespace would be a directed graph, with cycles. Since unix operating systems are responsible for deleting unused data in file systems (garbage collection), any algorithm used for that purpose has to satisfy strict requirements, for CPU and memory use, but more importantly for reliability. The algorithm must not fail the deletion unless the system is OOM, and it must always free the file's resources. Reference counting has always been used for this task. It's been awhile since I took graph theory, and I got a C in it anyway, but the algorithms I have seen for determining graph connectivity end up traversing to every node in the graph in the worst case. This means that the file system would potentially have to read in the directory data for every link to every file in the system, for a single deletion operation. If the issue is really the matter of removing the acyclic property, wouldn't that be solved by requiring that the file contains no non-dynamically generated subdirectories? That way, the graph is still acyclic, making reference counting work again. I had understood that a big part of the issue with file-as-directory was the same as the issue with hard links to directories. Which I thought is that if you move one directory into another, you can lose the connection to the root of the filesystem -- I.E. ../../.. etc is no longer guaranteed to get you to / -- And of course this also means that there is no way to get from / to your freshly moved files, and you've just lost a chunk of disk space to complete inaccessibility. fsck would have to be made smarter specifically to be able to figure that one out, too. Forbiding subdirectories in file-as-directory solves that problem too, because a normal file cannot be moved into anothers' file-as-directory. You could make it lose its meta-data, I suppose, but that's potentially lossy, which mv isn't allowed to be. Actually, when I had first read about file-as-directory, I had assumed that the content was either dynamically generated from the on-disk metadata (like uid, gid, etc) or stored as a sideband type stream in the file itself, like some of the MAC OS file systems (and others) do, or generated via a plugin connecting to user-space, for ID3 tags on mp3 files and other information which can easily be obtained from the file itself.
Re: Reiser4 documentation specifically current status of repacker, compression, and semantics
On Monday 24 October 2005 14:44, Edward Shishkin wrote: John Gilmore wrote: There is a 'compress' directory in in my reiser4 source tree. So is the compression plugin done? Is my disk/files compressed? not yet, this is under construction.. And if it was/wasn't how would I know the difference, and change it? I guess reiser4progs will be able to dump the attributes then Edward. I thought the ability to muck around with the plugins assigned to a particular file was implemented as file-as-directory? I've tried to re-enable that ability, but haven't been able to get it to work. How are plugin/file relationships handled without it?
Re: Reiser4 documentation specifically current status of repacker, compression, and semantics
On Monday 24 October 2005 19:08, Edward Shishkin wrote: John Gilmore wrote: How are plugin/file relationships handled without it? For the first time there will be an option in mkfs to assign a file plugin for regular files per superblock. Then (if everything will be okay) we will granulate the relationship. Edward. So the first beta of compression will only have the option to control it at mkfs time? And then later on, compression control and status information will be available on a per-file (or per-directory) basis via a user-space tool operating on mounted filesystems? Which tool will also be extendable to control and give status information (where applicable) on other types of plugins? Is there a page that I could go to - some sort of Major feature changes page - to see what the current status of the filesystem is? To answer such questions as Is compression ready for beta? and what the heck happened to file-as-directory? and I saw mention of a test of the reiser4 repacker, and also mention of compile problems with repacker.c, which definitely doesn't exist in MY version of resier4 - what gives? (though I've concluded that that last one must be somebodies copy of alpha software...)
Reiser4 documentation specifically current status of repacker, compression, and semantics
There is a 'compress' directory in in my reiser4 source tree. So is the compression plugin done? Is my disk/files compressed? And if it was/wasn't how would I know the difference, and change it?
