reiserfsck 3.6.19 fails to rebuild-tree badly broken fs
Hello list, I have a fs that has been broken by controller cache in combination with multiple power outages. It would mount, and many files were accessible, while some were not. I decided to clean it up so I could safely start writing to the filesystem again, and set out to fsck, but did not get far. I started with check, which old me to rebuild-tree. Attached is the log from rebuild-tree. I've run rebuild-tree several times and always get the same result, except for the number 91163 after left which can change between runs. Block 33147 never changes however. I don't have a dd backup of the fs for forensic purposes, but I'll gladly try any suggestions you can offer on the real fs. At the very least I would like to get the fs back to mountable state and copy some more files off of it before making a new fs. Do I only need to change the root inode (to 0?) or is there more to it? I'm not afraid of hacking up some utilities to fix/debug the fs, just let me know how to proceed. (Does reiserfsck want to deal with this broken fs:s?) Should this be a $25 support question? Thanks in advance, and thanks for the work you've put into ReiserFS! //Peter --8-- # reiserfsck --rebuild-tree -l sdb2.log /dev/sdb2 reiserfsck 3.6.19 (2003 www.namesys.com) * ** Do not run the program with --rebuild-tree unless ** ** something is broken and MAKE A BACKUP before using it. ** ** If you have bad sectors on a drive it is usually a bad ** ** idea to continue using it. Then you probably should get ** ** a working hard drive, copy the file system from the bad ** ** drive to the good one -- dd_rescue is a good tool for ** ** that -- and only then run this program. ** ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to reiserfs-list@namesys.com, ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** * Will rebuild the filesystem (/dev/sdb2) tree Will put log info to 'sdb2.log' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes Replaying journal.. Reiserfs journal '/dev/sdb2' in blocks [18..8211]: 0 transactions replayed ### reiserfsck --rebuild-tree started at Mon Jan 8 00:00:54 2007 ### Pass 0: Loading on-disk bitmap .. ok, 33991146 blocks marked used Skipping 9280 blocks (super block, journal, bitmaps) 33981866 blocks will be read 0%20%40%60%80%100%left 0, 5675 /sec r5 hash is selected Flushing..finished Read blocks (but not data blocks) 33981866 Leaves among those 91514 - corrected leaves 12 - leaves all contents of which could not be saved and deleted 26 pointers in indirect items to wrong area 6 (zeroed) Objectids found 281451 Pass 1 (will try to insert 91488 leaves): Looking for allocable blocks .. finished 0% left 91163, 325 /sec The problem has occurred looks like a hardware problem (perhaps memory). Send us the bug report only if the second run dies at the same place with the same block number. build_the_tree: Nothing but leaves are expected. Block 33147 - unknown Aborted # --8-- sdb2.log: ### Pass 0 ### block 33148: The number of items (12) is incorrect, should be (0) - corrected block 33148: The free space (0) is incorrect, should be (4072) - corrected block 33158: The number of items (32784) is incorrect, should be (5) - corrected block 33158: The free space (176) is incorrect, should be (2176) - corrected pass0: vpf-10410: block 33158, item 3: Wrong order of items - change the type of the key [2097154 7211 0x4 ??? (15)] to StatData pass0: vpf-10430: block 33158, item 2: Wrong order of items - change the dir_id of the key [4098 67112490 0x5 DRCT (2)] to 2 pass0: vpf-10560: block 33158, item 2: Wrong order of items - change the object_id of the key [2 67112490 0x5 DRCT (2)] to 7338 pass0: vpf-10170: block 33158: item 2: Wrong order of items - the item 2 7338 0x5 DRCT (2), len 240, location 2644 entry count 65535, fsck need 0, format new fixed to 2 7338 0x1 DRCT (2), len 240, location 2644 entry count 65535, fsck need 0, format new pass0: vpf-10460: block 33158, item 3: Wrong order of items - change the dir_id of the key [2097154 7211 0x0 SD (0)] to 2 pass0: block 33158, item 2 (upper): Item [2 7338 0x1 DRCT (2)] is out of order - deleted pass0: block 33158, item 1 (upper): Item [2 7338 0x0 SD (0)] is out of order
Re: r4 observations
On Fri, 22 Sep 2006 14:02:44 +0200, Thomas Kuther wrote: On Thu, 21 Sep 2006 11:18:41 + (UTC) Peter [EMAIL PROTECTED] wrote: On Thu, 21 Sep 2006 15:02:01 +0400, Vladimir V. Saveliev wrote: Hello On Wednesday 20 September 2006 22:47, Peter wrote: I booted from a non-reiser4 partition in order to make a backup of my main / which was a r4 partition. After the backup, I unmounted the drive explicitly, then rebooted. I did not use the backed up drive for anything except tar-ring its files. On next boot to the r4 / partition, all kinds of file not found errors occurred. I booted again to my non-r4 partition, and ran fsck.reiser4 --check -y there were fatal errors on my r4 /. The backup was fine. I downgraded back to reiserfs which does not exhibit this problem. I have not experienced any problem with other r4 partitions. Just /. /home, /tmp, /src, etc. are fine. Unfortunately, I don't have time to keep wondering where the problem is or why. Perhaps it's the kernel or the init scripts. Nonetheless, the instability of whatever the problem is is unnerving. Please provide information about which kernel and which reiser4 did you use. Am I correct that you were trying to run gentoo on reiser4? Yes, Vladimir. When making the backup, I was not running Gentoo at all. I was running Slackware 10.2. I booted into Slackware with the beyond patchset (ck superset) based on 2.6.17.11 with the reiser4 2.6.17-3 patch. All of the work on backup mount and unmount was on Slackware. It was when rebooting back into Gentoo (with the init and base layout which DID NOT CAUSE a boot problem) that the fatal errors occurred. Interestingly, and maybe this is helpful, only the / partition seems to be affected. I have observed no problematic behavior with any of the other three partitions I used r4 for. In fact, even though I downgraded / to reiserfs3, the other r4 partitions work fine. Please let me know if I can provide additional information. 2.6.17-beyond includes -ck1 which includes fcache which is totally evil for reiser4. The old fscache patch in .17-ck1 tends to kill reiser4 /usr here (I assume your /usr is in / then) when rebooting/umounting the partition. Also newer -mm is evil and umount on /usr completely fails here, i have to hard reset the box, but here the partitions stay alive, with the older fcache i had to --build-sb and --fix it, and then remerge all stuff that was lost. Oh and this happens even i didn't enable fcache in kernel config, just the existance of the patch is enough. Strangely it's reproducably and only /usr. So if you run the /usr folder on a reiser4 partition, do NOT use -mm, or at least good luck trying to break out fcache. Reversing the fcache patch from -beyond is easy, just get the broken-out from -ck and patch -R it. HTH Tom Thanks for the tip, Tom! You're the second person (unless you're the same one on the gentoo forums) to mention this. I suspect, with your use of the word evil, you're the same! :) Nonetheless, I lost patience, and did not want to be in a changing beta situation as kernels, patches, etc. are all changing at different rates of speed. I found this bug both annoying and disconcerting. I have one r4 partition left, and since it's not / there appear to be no problems. I assume the various parties involved are all aware of this? -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: r4 observations
On Fri, 22 Sep 2006 14:49:30 +0200, Thomas Kuther wrote: snip... Yes, that was me :) As i seemed to be the only person hitting that bug, i didn't mention it here on the list nor on LKML. I just warned the users in the -beyond and -no sources threads some releases ago ... .17-beyond1 thread IIRC... and stopped using fcache patched kernels, because i like my /usr/portage flying on a reiser4 partition ;) Regards Tom I passed this along to the beyond maintainer fwiw. Still a very disturbing problem. And, it appears, you're not alone. Well, not too alone anyway! -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: partition cannot be mounted from grub
On Thu, 21 Sep 2006 08:16:26 +0200, Huub wrote: Huub wrote: Hi, Using Suse 10.1, I had a spontaneous reboot after which I got a fast running screen full of reiserfs messages. Another (inflicted) reboot and Grub gives error 17, meaning it cannot mount a recognized partition. Using the boot-dvd, I selected Rescue, and logged in as root but I have no idea how to proceed trying to save my system. Help is appreciated. Thanks, Huub Correction: errormessage means: This error is returned if the partition requested exists, but the filesystem type cannot be recognized by GRUB. I.e. GRUB doesn't recognize it's a ReiserFS partition... Two suggestions. 1) I have found that grub files in r3 have to be copied using -o notail on the mount command. If you do not use notail, the files are not able to be read by grub. 2) do not try and create a grub boot while in the r3 partition. Make sure the partition is unmounted, then boot from a live cd and run grub from there. e.g. in your case, with /dev/hdc2 being your root root(hd2,1) # at this point grub should respond that filesystem is reiserfs setup(hd2) # at this point, grub should install the boot loader. # unless hdc is NOT your boot disk. If not, then use hd0 or 1. If not, try recopying /boot/grub by doing this from the live cd. mount /dev/hdc2 /mnt/wherever -o notail,rw,[noatime] cd /boot cp -rP grub grub2 rm -fr grub mv grub2 grub then umount it and try the grub commands above. I have had lots of woes with grub and reiser 3/4. This seems to work for me. -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: r4 observations
On Thu, 21 Sep 2006 15:02:01 +0400, Vladimir V. Saveliev wrote: Hello On Wednesday 20 September 2006 22:47, Peter wrote: I booted from a non-reiser4 partition in order to make a backup of my main / which was a r4 partition. After the backup, I unmounted the drive explicitly, then rebooted. I did not use the backed up drive for anything except tar-ring its files. On next boot to the r4 / partition, all kinds of file not found errors occurred. I booted again to my non-r4 partition, and ran fsck.reiser4 --check -y there were fatal errors on my r4 /. The backup was fine. I downgraded back to reiserfs which does not exhibit this problem. I have not experienced any problem with other r4 partitions. Just /. /home, /tmp, /src, etc. are fine. Unfortunately, I don't have time to keep wondering where the problem is or why. Perhaps it's the kernel or the init scripts. Nonetheless, the instability of whatever the problem is is unnerving. Please provide information about which kernel and which reiser4 did you use. Am I correct that you were trying to run gentoo on reiser4? Yes, Vladimir. When making the backup, I was not running Gentoo at all. I was running Slackware 10.2. I booted into Slackware with the beyond patchset (ck superset) based on 2.6.17.11 with the reiser4 2.6.17-3 patch. All of the work on backup mount and unmount was on Slackware. It was when rebooting back into Gentoo (with the init and base layout which DID NOT CAUSE a boot problem) that the fatal errors occurred. Interestingly, and maybe this is helpful, only the / partition seems to be affected. I have observed no problematic behavior with any of the other three partitions I used r4 for. In fact, even though I downgraded / to reiserfs3, the other r4 partitions work fine. Please let me know if I can provide additional information. -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: partition cannot be mounted from grub
On Thu, 21 Sep 2006 10:02:19 -0400, Jeff Mahoney wrote: snip... I'm curious what version of grub caused you to come to these conclusions. Grub has been using the REISERFS_IOC_UNPACK ioctl for ages. It's the same thing lilo uses and causes those files to only use indirect blocks (ie: no tails). If you've been running into this problem with any recent version of grub, it's a bug. - -Jeff 0.94/0.97 (not grub2). Just happened yesterday as a test. I also found it impossible to run grub in its /boot/grub mounted partition. I had to boot off another partition or a live cd and run grub from there. I did not find this trouble with r4 however. Just my experience however. -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
r4 observations
I booted from a non-reiser4 partition in order to make a backup of my main / which was a r4 partition. After the backup, I unmounted the drive explicitly, then rebooted. I did not use the backed up drive for anything except tar-ring its files. On next boot to the r4 / partition, all kinds of file not found errors occurred. I booted again to my non-r4 partition, and ran fsck.reiser4 --check -y there were fatal errors on my r4 /. The backup was fine. I downgraded back to reiserfs which does not exhibit this problem. I have not experienced any problem with other r4 partitions. Just /. /home, /tmp, /src, etc. are fine. Unfortunately, I don't have time to keep wondering where the problem is or why. Perhaps it's the kernel or the init scripts. Nonetheless, the instability of whatever the problem is is unnerving. -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: reiser4: mount -o remount,ro / causes error on reboot
On Sun, 17 Sep 2006 21:45:29 +0300, Jussi Suutari-Jääskö wrote: Peter wrote: On Sun, 10 Sep 2006 17:01:18 +, Peter wrote: this bug was also reported on gentoo wrt the newer baselayout. Indications are it may be a r4 issue, although no one seems to know why! http://bugs.gentoo.org/show_bug.cgi?id=144093 I have the same problem, segfault on boot if the system was shutdown properly. For me the trouble appeared after upgrading to glibc-2.4, there's no problem at all with glibc-2.3.6. On my system (amd64 platform with 64-bit gentoo installation) it doesn't have anything to do with baselayout, just the glibc. The fellow who originally posted the bug does not use r4 nor does he use glibc 2.4. http://bugs.gentoo.org/show_bug.cgi?id=147622 Yet, he has the problem we all are enduring. :( -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: [PATCH] reiser4: fix readv
On Thu, 14 Sep 2006 11:28:29 +0200, Christian Trefzer wrote: Hi Vladimir, this fixes a bunch of random user space oddities for me. Just patched 2.6.18-rc6-mm2 with this, and some annoyances I could not trace back to a change in user space just went bye-bye. Thanks a lot! Chris This does not patch against 2.6.17-3 patchset however. Any possibility this may be related to some startup issues as noted on other threads here? Thx -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: reiser4: mount -o remount,ro / causes error on reboot
On Sun, 10 Sep 2006 17:01:18 +, Peter wrote: this bug was also reported on gentoo wrt the newer baselayout. Indications are it may be a r4 issue, although no one seems to know why! http://bugs.gentoo.org/show_bug.cgi?id=144093 -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: reiser4: mount -o remount,ro / causes error on reboot
On Tue, 12 Sep 2006 21:48:16 -0500, David Masover wrote: snip... Sorry to report this as an r4 bug, although it's interesting to note that the 1.12.4 baselayout did NOT cause this problem in reiserfs3.6 Mine was doubtlessly a Reiser4 bug, as it resulted in either an oops or a panic, I'm not sure which. I think it was an oops, and after that oops, the disk is inaccessible. Since it's a root FS, this is a problem! The init scripts should not be able to cause this, no matter how buggy they are. That's good to know...I guess :) I haven't done this already, because everything works now, and no one's asked me to yet. I find this curious, don't you? I'm wondering if kernel preemption settings and anticipatory read ahead settings could play a role? Mine are CONFIG_PREEMPT=y and CONFIG_DEFAULT_IOSCHED=CFQ. One other thing of note is that my kernel has Jens Axboe's iosched-rollup-2.6.17.4-2 patch. My reiser4 patch is 2.6.17-3. The iosched patch removed a lot of dma messages in the kernel log (not disk errors). Anyway, whatever changed in my downgraded baselayout, at least it's not causing reiser4 to hiccup. Curious problem in that it only occurs immediately after a shutdown, not after the error and a reboot from a maintenance prompt. -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: reiser4: mount -o remount,ro / causes error on reboot
On Wed, 13 Sep 2006 14:49:05 +0400, Vladimir V. Saveliev wrote: snip... I still think that the problem is in reiser4. When the system fails on boot it usually outputs something which may help to understand the problem. Do you see anything like that on faulty startups? You can use either serial or network console to catch kernel output. Well, the output is from the gentoo rc script and from the imported functions.sh script. It showed two segfaults. It occured afaikt when the dmcrypt addon is called. It uses the start-stop-daemon program. I tried taking a photo of it, but it was blurry and the flash obscured it. I backtracked all the way from the errant baselayout and until 1.11.15-r3 none of the 1.12 series of baselayouts worked. Too bad I can't get gpm to run otherwise I could capture the output. However, the only output are lines from the script with uninitialized variables -- basically the source of the scripts. No dumps, stack traces, or panics. -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: reiser4: mount -o remount,ro / causes error on reboot
On Wed, 13 Sep 2006 14:49:05 +0400, Vladimir V. Saveliev wrote: all snip. Here is a screen shot I posted along with the bug report on this: http://bugs.gentoo.org/attachment.cgi?id=96874action=view . I am sorry the pic is a little blurred, but I had battery trouble. There are two segfaults that occur, 2399 and 2524 and the text that is printed is from line 390 of rc and line 181 of functions.sh. As you can see, there are no panics or dumps and it appears that for whatever reason, the init scripts just cannot continue. However, the reboot (ctrl-d), the scripts execute fine. As I noted previously, the error occurs in all unmasked 1.12 base layout files. 1.11.15-r3 works fine. -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: Relocating files for faster boot/start-up on reiser(fs/4)
On Wed, 13 Sep 2006 14:51:39 -0600, Quinn Harris wrote: Thoughts? Yes. Why on earth would you do this? By copying the files and renaming and hardlinking them is nothing a sysadmin would ever do. Just by copying you are allowing reiser to optimize the dir. You're trying to duplicate what a tree-based design does automatically. Moreover, remember that reiser packs files into clusters so that you may read more than just your one file from time to time which could end up adding time to your test. If reiser needs speedup it certainly won't be done by renaming files! JM$0.02 -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: reiser4: mount -o remount,ro / causes error on reboot
On Sun, 10 Sep 2006 17:01:18 +, Peter wrote: all snip... To Vladimir and David: This appears to be a nasty gentoo issue. After perusing the forums and bugzilla, it appears that we are not alone in having difficulties with the baselayout. Nonetheless, as the reporter did, I downgraded baselayout from 1.12.4-r7-r7 to 1.11.15-r3 and the reboot problem I noted went away. It is interesting to note that it may be a C program startstop-daemon.c that may be the culprit. I don't expect much help from the gentoo devs since they won't support reiser4, but thought I would throw this out. Sorry to report this as an r4 bug, although it's interesting to note that the 1.12.4 baselayout did NOT cause this problem in reiserfs3.6 HTH -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: reiser4: mount -o remount,ro / causes error on reboot
On Tue, 12 Sep 2006 21:10:08 +, Peter wrote: On Sun, 10 Sep 2006 17:01:18 +, Peter wrote: all snip... To Vladimir and David: This appears to be a nasty gentoo issue. After perusing the forums and bugzilla, it appears that we are not alone in having difficulties with the baselayout. Nonetheless, as the reporter did, I downgraded baselayout from 1.12.4-r7-r7 to 1.11.15-r3 and the reboot problem I noted went away. It is interesting to note that it may be a C program startstop-daemon.c that may be the culprit. I don't expect much help from the gentoo devs since they won't support reiser4, but thought I would throw this out. Sorry to report this as an r4 bug, although it's interesting to note that the 1.12.4 baselayout did NOT cause this problem in reiserfs3.6 HTH http://bugs.gentoo.org/show_bug.cgi?id=147274 I have learned that the new baselayout caches some services during startup. While this does not explain why all works fine on a reboot and not on the first boot, perhaps there is something not being cached properly? -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: reiser4: mount -o remount,ro / causes error on reboot
On Mon, 11 Sep 2006 13:10:54 +0200, Sander Sweers wrote: snip... There was a bug in baselayout which caused partition (except /) not to remount ro properly. The bug number is 131001 [1], is this your problem? Greets Sander 1: http://bugs.gentoo.org/show_bug.cgi?id=131001 Thank you, I read that. My version of baselayout has that fix, but that does not appear to be the problem. I think it has more specifically to do with the way the remount option is affecting the reiser4 fs. And, only / is left when the remount,ro part comes anyway. Everything else is unmounted by that time. -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: reiser4: mount -o remount,ro / causes error on reboot
On Mon, 11 Sep 2006 11:30:39 +0400, Vladimir V. Saveliev wrote: snip... Sorry, I am confused. In the first mail you said: On reboot or after a poweroff, root does not mount properly, and after some modules are loaded, there are segfaults when running init scripts. This looks like you have problems on startup. Would you, please, describe the sequence of operations which leads to the problem with more details. I should have written that it occurs always after a normal shutdown or reboot. On initial startup, the error occurs. Then, after CTRL-D, the system reboots and all is fine. Then, after the day, normal shutdown, then abnormal startup. Some more information, I looked at the output from the final mount -v -n -o remont,ro / command, it appears perfectly normal. However, I am not sure it is working normally! -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: reiser4: mount -o remount,ro / causes error on reboot
On Sun, 10 Sep 2006 17:01:18 +, Peter wrote: Using: gentoo kernel 2.6.17.11 with beyond patchset reiser patch 2.6.17-3 reiser4progs 1.0.5 update... Transferring / to a reiser3 partition removes this problem. Shutdown and startup proceed normally. I am using util-linux-2.12r with gentoo patches -r4. This was updated on 9/4/06. I am thinking I will downgrade to -r3 and see if that removes the problem. -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
reiser4: mount -o remount,ro / causes error on reboot
Using: gentoo kernel 2.6.17.11 with beyond patchset reiser patch 2.6.17-3 reiser4progs 1.0.5 At the end of the gentoo shutdown script is a short function which remounts / as ro. sync; sync sleep 1 . # ${x} = / mount -n -o remount,ro ${x} /dev/null On reboot or after a poweroff, root does not mount properly, and after some modules are loaded, there are segfaults when running init scripts. The init scripts prompts for the root password for maintenance. Simply typing exit forces an unmount of / and a reboot. After this, everything is normal. On a normal shutdown, this error cycle begins anew. Something has changed in the gentoo shutdown sequence as this error is new to my system. However, gentoo does not support reiser4 as it is experimental. I wanted to report this. Remount is required since with the halt script running, the root cannot be unmounted. After halt.sh, reboot -idpk is called or shutdown. My hds are IDE /dev/hda: Model=Maxtor 6Y200P0, FwRev=YAR41BW0, SerialNo=Y65WFMPE Config={ Fixed } RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=57 BuffType=DualPortCache, BuffSize=7936kB, MaxMultSect=16, MultSect=16 CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=268435455 IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120} PIO modes: pio0 pio1 pio2 pio3 pio4 DMA modes: mdma0 mdma1 mdma2 UDMA modes: udma0 udma1 udma2 udma3 udma4 udma5 *udma6 AdvancedPM=yes: disabled (255) WriteCache=enabled Drive conforms to: (null): ATA/ATAPI-1 ATA/ATAPI-2 ATA/ATAPI-3 ATA/ATAPI-4 ATA/ATAPI-5 ATA/ATAPI-6 ATA/ATAPI-7 * signifies the current active mode -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: reiser4: mount -o remount,ro / causes error on reboot
On Sun, 10 Sep 2006 15:12:00 -0500, David Masover wrote: Peter wrote: Using: gentoo kernel 2.6.17.11 with beyond patchset reiser patch 2.6.17-3 reiser4progs 1.0.5 At the end of the gentoo shutdown script is a short function which remounts / as ro. There's also one in the Gentoo startup script, which attempts to remount / ro, then remount it rw. I commented that out, because it was causing similar problems. I figure if it runs sync when it shuts down, that's good enough. The errors I note only occur on shutdown (halt.sh) not startup. Do you think it could be an IDE timing thing similar to what was described on another thread on this ml? What's interesting is that this problem is recent and I am trying to look back and see what system-level packages were updated recently (I just converted to the 2006.1 profile before this occured and that caused a lot of programs to recompile. A week earlier, it was gcc-4.1.1. I know base layout was updated recently). Maybe something in mount changed? The shutdown scripts look the same. Something is left unhinged somewhere. Glad I was not hallucinating! Thanks for confirming for me. Still, it's an annoying problem, I think it's a kernel oops. Namesys, what kind of information would be helpful? Yes, it's annoying and disconcerting at the same time. If it was a kernel oop then, wouldn't it have shown itself earlier? I -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: FEATURE Req: integrate badblocks check into fsck.reiser*
On Thu, 07 Sep 2006 12:35:06 -0500, David Masover wrote: all snip. All I was getting at originally was to make fsck.reiser4 behave similarly to other fsck/mkfs programs so that it _could_, when requested, check a disk/partition for bad blocks. Despite the improved capabilities of modern drives, sh*t happens. And I, for one, would like the option of knowing before formatting a new partition if it's OK to do so. Sure, I can run badblocks myself, but this is counter to what I do with other fsck programs. A wrapper script would work fine, and I suppose, some sort of pipe output. However, I'm just looking at this from a userland perspective, not that of a sysadmin. Obviously, a sysadmin would not put any disk into production that has not been carefully tested. If this is superfluous, that's OK. I can easily work around it, but it should be noted somewhere in a README that reiserfs/4 does not do bad block checking during format and only a quick format is done. -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: reiser4 corruption on initial copy
On Tue, 05 Sep 2006 13:30:54 +0400, Vladimir V. Saveliev wrote: hello On Monday 04 September 2006 18:32, Peter wrote: On Fri, 01 Sep 2006 11:02:58 +, Peter wrote: I recently copied over my / partition from a reiserfs to a reiser4 partition. This may be user error, but I wanted to report it anyway. Booting off a live cd- I did the following. snip... I was able to duplicate the problem. Apparently, the Sabayon live cd does not unmount locally mounted partitions. That should not be a problem if sync is called after copy is completed. It looks like Sabayon live cd even does not care to call sync on reboot. which clearly explains a lot of the headaches I had when creating or moving partitions from w/in it! I'll forward this thread to upstream. -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: reiser4 corruption on initial copy
On Fri, 01 Sep 2006 11:02:58 +, Peter wrote: I recently copied over my / partition from a reiserfs to a reiser4 partition. This may be user error, but I wanted to report it anyway. Booting off a live cd- I did the following. snip... I was able to duplicate the problem. Apparently, the Sabayon live cd does not unmount locally mounted partitions. Thus, the reiser4 partitions are unclean. When I boot up to the newly created and copied partition, I get the errors I referred to earlier. By manually umounting the partitions I create or copy from -- including /tmp partitions or /var partitions. I do not get the boot time errors. HTH -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: reiser4 corruption on initial copy
On Fri, 01 Sep 2006 17:35:29 -0500, David Masover wrote: Peter wrote: 2) I did run badblocks on the dest, and it was clean. 3) I am using the patch from 2.6.17.3 and in my kernel, I have full preempt and cfq scheduling. What about the kernel on the livecd? Anticipatory Voluntary Plus, it is smp, so there are some additional options checked. Should I have preempt=none with reiser4? -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
wrt: checking reiserfs/4 partitions on boot
On the namesys.com FAQ page, it is recommended that 0 0 be placed at the end of the fstab lines for reiserfs partitions. I have two questions: 1) does this recommendation also apply for reiser4? 2) why is this recommendation made? Is it unnecessary to routinely check reiser partitions? I understand that in the event of an abnormal shutdown, fsck will be forced, correct? Thx -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
reiser4 corruption on initial copy
I recently copied over my / partition from a reiserfs to a reiser4 partition. This may be user error, but I wanted to report it anyway. Booting off a live cd- I did the following. mount source partition -o ro,noatime mount destination partition -o rw,noatime # cd source partition # tar --one-file-system -cvf - | tar -C dest -xf - Tar went along fine. At the end, there was a very long pause, about a minute, after the last file (a symlink to /var) was copied. Then the prompt returned. On booting up, there were a stream of kernel errors, etc. So I booted another / and fsck destination partition. It found one error in a file, just before the last file (/var symlink) copied. I fixed it, it removed the file (/sbin/uname). Recopied it over from the source partition. After this, the partition works fine. I wanted to know: 1) did I copy over the stuff wrong? I also used cp -ax and had the same problems except this time the disk could not be repaired. 2) I did run badblocks on the dest, and it was clean. 3) I am using the patch from 2.6.17.3 and in my kernel, I have full preempt and cfq scheduling. Any feedback appreciated! Thx and keep up the good work. -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: reiser4 corruption on initial copy
On Fri, 01 Sep 2006 11:02:58 +, Peter wrote: # cd source partition # tar --one-file-system -cvf - | tar -C dest -xf - of course I had * for all files :) I also did the same with rsync -a /mnt/dest. I then thought perhaps the livecd did not properly unmount the local filesystems, so I issued the umount command. This took approximately three minutes. I then ran an fsck and it appeared fine. However, again on first boot, the kernel issued a slew of errors, mostly about not being able to access /proc (which was there). Then, again, after a reboot, I was OK except that one file /sbin/uname was bad again and needed to be fixed and replaced. I suppose I will try and see if I can tar up the reiser3 partition and restore it in a separate action. If you require additional information, please let me know. -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
FEATURE Req: integrate badblocks check into fsck.reiser*
Perhaps this has been mentioned before. If so, sorry. IMHO, it would be useful to integrate a call to badblocks in the fsck/mkfs.reiser* programs so that more thorough disk checking can be done at format time. Sort of like the option e2fsck -c. If this is added, the output could be fed immediately to the reiser format program and badblocks spared prior to filesystem use. JM$0.02 -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: FEATURE Req: integrate badblocks check into fsck.reiser*
On Fri, 01 Sep 2006 17:27:20 -0500, David Masover wrote: snip... both mkfs.reiserfs and fsck.reiserfs have -B option to accept list of bad blocks. We thought that should be enough. It really should. Why bother with a patch? Just write a wrapper script that runs badblocks and passes in the list to mkfs. It was just a thought from userland. My perspective was that a user, not a hard-boiled geek, might get lulled into a false sense of security but may not have the wherewithal to write a wrapper. If nothing else, when the final doc is written (did I say final?:)), it should include a notice about not running badblocks. -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: Repacker and compression plugin
On Monday 27 March 2006 23:25, Hans Reiser wrote: (none of which benefit reiser4 much). Mainline will benefit R4 much. Good luck! I know you guys can do it! -p -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada
Re: funding - The Rational Street Performer Protocol
Lex Lyamin wrote: http://www.logarithmic.net/pfh/rspp idea is good, now if only we could come up with implementation, which looks like it will only work with with some liquid nitorgen cooling, nonetheless ( impossible ). Here is a Python implementation for a (piecewise-linear monotonic) pledge function http://www.logarithmic.net/pfh-files/rspp/rspp.py (calculation of the maximal sum). Here is another: http://thecircle.org.au/compute_contribution.py But the main issue is explaining the idea, advertising and getting the pledges, not the calculation. A web page would help but just putting up a web page is not enough, it needs a lot of work trying to convince parties who may benefit that it makes sense for them (e.g. Linux distributors). (A much simpler implementation (and idea) is PledgeBank http://www.pledgebank.com/ with much less flexibility. The advantage is that the site already exists.)
funding - The Rational Street Performer Protocol
Hans, Here is an interesting idea. Could it be a possible model for funding software development? http://www.logarithmic.net/pfh/rspp What do you think? Peter
Re: BitKeeper
Hans, I'd like to run a mirror of the R4 dev tree so that the R4 dev wiki can reference specific files and versions of files as well as the changelog. You mentioned R4 development is done using git these days. Is that repository public, or at least publicly mirrorable? -p On Tuesday 07 March 2006 10:31, Hans Reiser wrote: flx, please fix bitkeeper related web pages. Hans Yoanis Gil Delgado wrote: On Tuesday 07 March 2006 11:23, Amit Vijairania wrote: Hi! I want to contribute to Reiser4, and would like to dowload latest development source. Namesys.com gives directions for using BitKeeper repositories. Is this the only way to download latest code? Thank you. Amit if you want to contribute to reiser4, maybe you would like to participate in the plugin fest. -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada pgpvqQI5Lt8go.pgp Description: PGP signature
Re: Plugin Patch (was Re: creating live virtual files by concatenation)
On Tuesday 28 February 2006 12:07, Yoanis Gil Delgado wrote: On Tuesday 28 February 2006 01:24, Peter van Hardenberg wrote: We most take the advance.I suggest to all people interested in this to spent a full weekend creating such a plugin. There is a possibility of failure but. we will gather enough question for the people at namesys and they can share some ligth. Then we spent another weekend and so on... Yoanis, this is a great idea! We can collaborate via IRC and the wiki and share our discoveries. I will join this project. Who else will? -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada pgpFToE68EfrS.pgp Description: PGP signature
Re: Plugin Patch (was Re: creating live virtual files by concatenation)
On Tuesday 28 February 2006 12:04, Hans Reiser wrote: Maybe we can put a button on our webpage labeled reiser4 wiki maintained by pvh. There is no chat equivalent of mailto: is there? Maybe we should also have a button labeled chatrooms which contains a page describing where to find our chatrooms, etc. An excellent idea. I know there is already a Reiser IRC channel on OFTC. The URL irc://irc.oftc.net/reiser4 is understood by some programs. I have added the link to my wiki front page. If involvement grows, I will probably improve the Reiser4 wiki's hosting situation, at which point I will redirect the URL automatically and post the new URL here. Tell me, do you have an public access SVN or CVS repository I could link the wiki to for source references? -pvh -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada pgpvjj1EeFuYe.pgp Description: PGP signature
Re: Plugin Patch (was Re: creating live virtual files by concatenation)
On Tuesday 28 February 2006 16:03, you wrote: On Tuesday 28 February 2006 17:24, you wrote: No i don't have SVN or CVS. Also i think it will be a nice feature of the wiki if it explains how to setup a development enviroment. It does, thanks to Ryan Nordman's efforts. (Further additions, corrections, and alternate configurations are welcome.) -p -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada pgpjXW7wI1U9o.pgp Description: PGP signature
Plugin Patch (was Re: creating live virtual files by concatenation)
Hans, you've said that these kinds of plugins should be something a weekend warrior could tackle. Our group had a real stab and dumped hundreds of man hours into the project with little effect. Admittedly, we were not experienced kernel hackers, but we were all comfortable in low-level C and quite happy to read source. I request that a simple plugin be maintained as a standalone patch to Reiser4. Ideally, there would be a small set of these plugins demonstrating how to create a new plugin which operates within the existing disk structure, and one that extends the on-disk format in a safe way. This would allow interested parties to see in isolation what a Reiser4 plugin looks like and would further provide a conceptual grappling point for the development of a new plugin. I have been getting requests for just such a plugin to be added to my reiser4 developer's wiki (http://pvh.ca/trac/wiki/reiser4) at a rate of about one every two months. A few successful third-party plugins would hopefully increase interest in this. I realise you and your team are up to your necks in serious work on hardcore lowlevel material, but I believe a brief diversion of some of your resources would provide a significant reward. Right now, the cost-of-entry appears to be set too high for developers outside your team to approach the project. If this information is already out there somewhere, great. I will integrate it in the R4DevWiki and answer questions as best I can. If anyone out there disagrees with me about the current difficulty of producing even a simple plugin, let them prove me wrong with a patch. -pvh -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada pgpmtn3bbb9eH.pgp Description: PGP signature
Re: Reiser4 default in Underground Dekstop distribution
Marcel (Hilzinger), Did you write this Linux Magazine article? Could you give some details on what the problem was? Peter On Sunday 26 February 2006 17:34, Hans Reiser wrote: There are so little details that it is hard to know what the problem they had was. It does sound though like we need to smooth out the commits. We have deferred that for a while. I know how we should do it, we just need to do it. Sigh, there are so many things like that. Probably fsync should be optimized first Hans Peter Foldiak wrote: The preview of the April (!?!) 2006 issue of Linux Magazine has an article http://www.linux-magazine.com/issue/65/Super_Underground_Desktop.pdf saying that Reiser4 is the root filesystem for the distribution Underground Desktop http://www.ludos.org/ but says it failed to live up to the expectations we had for Reiser4, blaming Reiser4 for some performance problems. Not much detail is given. It also says that Grub does not support Reiser4, so LILO is used. (Is that true?) Do you know of any other distributions that comes with Reiser4 by default root file system? Peter
Reiser4 default in Underground Dekstop distribution
The preview of the April (!?!) 2006 issue of Linux Magazine has an article http://www.linux-magazine.com/issue/65/Super_Underground_Desktop.pdf saying that Reiser4 is the root filesystem for the distribution Underground Desktop http://www.ludos.org/ but says it failed to live up to the expectations we had for Reiser4, blaming Reiser4 for some performance problems. Not much detail is given. It also says that Grub does not support Reiser4, so LILO is used. (Is that true?) Do you know of any other distributions that comes with Reiser4 by default root file system? Peter
Re: Reiser4 default in Underground Dekstop distribution
correction: Sorry, I now see on their site that: Underground Desktop 022 ... the filesystem is now reiserfs v3 instead of reiser4 - which seems not enough stable and fast - and the bootloader is grub instead of lilo. (It's funny, their April edition was already out of date in early February.) Peter
Re: creating live virtual files by concatenation
hi, I think it is a good idea but there has been quite a long discusion of this issue on the reiserfs-list (some of it also on lkml) in the last few years, with the latest posts in May 2005 I think, most of ithem with the file as directory subject . The idea was if you have e.g. a book directory/file (object) and chapters within it, then the default action of book object when read as a file would be to give a concatenation of the chapter objects within it. There should be ways of specifying non-default actions for the objects too. There has also been a lot a resistance here to this idea, as it is quite a radical change to the concept of the conventional unstructured, serial file. (I don't believe these are good reason not to try the idea at least.) Read those mails first. Peter On Saturday 25 February 2006 14:37, Maciej Soltysiak wrote: I have this idea about creating sort of a virtual file.
Re: creating live virtual files by concatenation
Maciej Soltysiak wrote: I can imagine quite a mess if I open a file that is really a view of several files and then start manipulating text in it across actual file boundaries that could blow up easily. Well, I meant that file to be read-only. Just a quick concatentated view for reading. The quick hack might be useful in certain situations. But the really interesting way to do it would be not to distinguish beween actual and non-actual files. All of them should be equally actual, it is just that containment (possibly even overlap) would be allowed. The tree structure used by a file system such as Reiser4 would make this very efficient with each sub-file corresponding to a key-range. Writing a chapter should change the book that the chapter is part of. That is what would make it really valuable. Of course it would have all sorts of implications (e.g. for metadata for each part) that need to be thought about, but it could be done properly, I think. Peter
Re: creating live virtual files by concatenation
Rik van Riel wrote: On Sat, 25 Feb 2006, Peter Foldiak wrote: sub-file corresponding to a key-range. Writing a chapter should change the book that the chapter is part of. That is what would make it really valuable. Of course it would have all sorts of implications (e.g. for metadata for each part) that need to be thought about, but it could be done properly, I think. What happens if you read the first 10kB of a file, and one of the chapters behind your read cursor grows? Do you read part of the same data again when you continue reading? Does the read cursor automatically advance? You should probably continue where you left off . Your idea changes the way userspace expects files to behave... Yes, I know.
reiser4: Request for minor change in max_name_len_cde()
The function max_name_len_cde() in reiser4/plugin/item/cde40.c in my opinion limits the file name length too strict. One finds: return tree_by_inode(dir)-nplug-max_item_size() - sizeof(directory_entry_format) - sizeof(cde_item_format) - sizeof(cde_unit_header) - 2; I think the -2 can be safely changed to -1 since the only extra byte that has to be taken into account is the terminating zero of the file name in case a longname key is encountered. The maximum file name length in the original case is 3976 bytes if the filesystem uses 4096 byte blocks. In this case the node header of the node containing the cde40 item reports 1 byte of free space. 4096 block size -28 sizeof(node40_header) -38 sizeof(item_header40) -2 sizeof(cde_item_format) -26 sizeof(cde_unit_header) -24 sizeof(directory_entry_format) -1 terminating zero of file name -1 ? - 3976 I changed the -2 to -1 and recompiled the kernel. Then I was able to create a file with a 3977 byte name. In this case the node header reports 0 bytes of free space. Best regards, Peter.
Re: metas
On February 10, 2006 07:08 am, Edward Shishkin wrote: Edward Shishkin wrote: Perhaps it got fixed when migrating to the new code for reiser4/vfs interface (Peter used the old one). Edward. I'll verify this some time soon. Good to hear! -p -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada
Identical files in reiser4 patch for linux-2.6.16-rc1-mm3
In linux-2.6.16-rc1-mm3 one finds the identical files reiser4/plugin/item/sde.h and reiser4/plugin/item/acl.h. Even their sentries are identical. Was that intended? Best regards, Peter.
Re: Authoring a versioning plugin
Hi Yoanis, good to see you're still pursuing this. On January 11, 2006 02:59 pm, Yoanis Gil Delgado wrote: This are the intentions: To write a versioning plugin that will allows the file system user to easily revert the files under versioning to a some previous state. The plugin will allow to revert the file state, based on revisions number and date modifications(and not sure about this one). There will be a special pseudo file named previous that will return the previous version of the file. The final result should allow to the the following actions: $ echo 1 myfile.txt (let's say we make this command at Wed Jan 11 16:53:55) $ echo 2 myfile.txt (let's say we make this command at Wed Jan 11 16:54:57) $ echo 3 myfile.txt (let's say we make this command at Wed Jan 11 16:55:59) Suppose you want the latest version, then you type: $ cat myfile/.../previous Some other content Or you want the n-th version, then you type: $ cat myfile/.../1 Some content $ cat myfile/.../2 Some other content $ cat myfile/.../3 This is going to clutter the ... directory rather a lot. Instead of adding more files into (which, by the way, is completely obscure) I would suggest you create a new pseudo directory. Perhaps: $ cat myfile/.^4/history/previous $ cat myfile/.^4/history/version/1 still not quite right, but at least it contains a bit more information about what the 1 refers to. Some other content Some more content $ Or the version nearest to some date, then you type: $ cat myfile.txt/.../Wed\ Jan\ 11\ 16:50 Some other content There are already userspace tools which can determine the file creation date. Just use those, instead of dealing with date parsing in kernel-space. Date parsing is a way, WAY more subtle problem than you want to deal with. To see a group that has spent some time on it, check out the Date::Parser for Perl. Using grep or find, or ls or whatever other tool will accomplish this in a much more thorough and Unix-consistent way and also save you a pile of coding time. Believe me, you're going to need it. Also , there will be an special attribute named under_versioning(or something like that), that will tell if the file is under versioning. This plugin will not track directories version, although it's a future plan(I think this should be mixed with some undelete plugin). I imagine that attribute should be $ echo 1 myfile.txt//plugins/versioning or $ echo everywrite myfile.txt//plugins/versioning Unfortunately, my experience is that you cannot use echo to change the data in the plugins/* pseudoplugins, even when it should be legal to do so. I just had a little ruby script that looked roughly like this: f = open pseudofile; f.write('newplugin'); f.close; Never had the time to figure out why that was necessary, but there it is. (There is a comment on the plugin-wiki gotchas section.) I'm planning to use a delta techniques for versioning storage (delta compression). The versioning will be at the write level. The versions will be saved in a special directory under the filesystem. I think the hard part is the one related to detecting the changes (a COW it's a possible solution, but i think it's to expensive). I'm thinking a possible solution will be detecting the bytes changing in each write and archiving then as the difference. This introduce some problems like : 1-) What happens if the file shrinks? 2-) What happens if the file grows ? I will send another email with a solution to this problems. This will not be easy, I look forward to seeing your solution. I've also plans to extent the documentantion of plugins creation in reiser4 with the experiences of this project. I'll be working in this plugin for more than 4 months. If you're interested you're welcome to the the team(just me right now :D ) Well, I have my own fish to fry, but I hope you will document your experiences on the Reiser4 programmer's wiki, currently housed at: http://pvh.ca/trac/wiki/reiser4 There is lots of important information there for new Reiser4 plugin developers, and it will continue to grow as time goes by. Well... I think this is all (for now :D ). Please let me know what you think. I would second Hans' suggestion about a /version/snapshot file which would essentially act like a cvs commit on that file. I'd suggest that there be two similar versioning plugins, one which automatically versions after each write, and one which only does it when explicitly asked to. See the fibration plugin type for an example of this. -p -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada
Re: reiser4: writing plugins
On December 22, 2005 05:37 pm, Jake Maciejewski wrote: Being able to use df or zfs list instead of du is a side effect. The real reason for the unusual abstraction is to have finer grained control over quotas, reservations, snapshots, checksums, compression, NFS sharing, etc. It seems to me that although it could be very valuable in certain situations, for general purpose consumption the performance hit would be too painful. If the data isn't kept up-to-date, it's not a lot better than not having it. You would always have to check if a deep child-directory is out of date before giving quota permission, which means still paying for those disk reads anyway. On the other hand, if the information is always kept correct, you're paying for a lot more writes and reads every time you touch a file... Also, the speed benefits for moving this in-kernel are not significant. The kernel also needs to essentially do a du -s -- the information is kept in the same place and has to be calculated the same way, which is by opening each file's inode and reading the file-size field. I wonder how Reiser4 would fare performance-wise on that kind of operation. The tree-based nature of the storage makes lookup() very fast, but I have noticed that the seekable hashed dir is described in the comments as a hack to work around the fact that real directories aren't actually seekable. This is why B-tree indexed databases maintain leafs in strict increasing order, which can cost a TON to maintain under certain kinds of load. -pvh -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada
Re: Reiser4 Plugin Developer Wiki
On December 19, 2005 12:39 pm, Hans Reiser wrote: I am sorry that this turned out to be more work than your budget allowed for, and I also regret that I think there was some deeper disagreement over the appropriate fundamental semantics, as the discussion with Peter Foldiak and I revealed. Hans Eh, we weren't being paid and we didn't have a customer, so we'll be alright in that regard. Still, I think the issue of user-defined attributes vs. file-as-dir still needs more work and I look forward to extending the debate some time. Let us assume that I am wrong about keeping attributes in a separate @ directory and continue from there. Perhaps after the resource fork implementation is complete the case for more options might be clearer. So, let us find a way to store standard files within files, as we can all agree on that! $ ./myScript Running myScript... $ cd myScript; ls; internal file contents $ Our existing research is still very much valid, but I don't think a pseudo-plugin is the way to implement this. Perhaps the best way to handle the issue would be to write a new directory plugin? Hans, I remember you described how you thought it would work in a sentence or two. Could you perhaps flesh out the idea to a paragraph or three? With that much guidance, I think we could go further without feeling like we are wasting effort. -pvh -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada
Re: reiser4: writing plugins
Danny, It's good to hear from you. The Reiser4 plugin system is somewhat subtle. Over the past four months our group has attempted to become the first group outside Namesys to create a Reiser4 plugin. We have had some various successes and failures and are hoping to document much of the experience for general consumption. -pvh On December 16, 2005 11:58 am, dannym wrote: Hi, So I want to start writing a small plugin to get a feel for the architecture... I wonder how to do that? I read that in order to write plugins, one has to recompile the kernel... which is ok, but that also means that it will affect everything and when I make a mistake, everything goes bad :) So I wondered if all you reiser4 developers have some kind of sandbox thing that just does the same with a library (without the kernel) and on an image file? Or do you use a virtual machine for that kind of stuff? As for what I thought would be a nice intro thing for me to write is: Adds a pseudo file tree-contents-size for directories. Pseudo file contains the du starting from this directory. What it should do is cache some kind of du-like value per directory. When you change a file size (or delete a file, add a file, ...) in the dir or one of it's subdirs, it shall set a dirty flag on the value. When the pseudo file is accessed, it shall calculate it anew, also updating the subdirs' pseudo files if so needed, and store it actually on disk... (could also be updated periodically without anyone requesting it, but that's just a detail) What do you think? Useful? Bad Thing? Already done? Too hard for a first try? cheers, Danny -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada
Why no pseudo_dir.c:get_parent()?
We need more pseudo_dir methods than you have defined, so we are extending the model slightly. In this process, we discovered this: /* pseudo files are not serializable (currently). So, this should just return an * error. */ reiser4_internal struct dentry * get_parent_pseudo(struct inode *child) { return ERR_PTR(RETERR(-ENOTSUPP)); } Please comment on why this returns -ENOTSUPP instead of returning the host file/pseudo file which is the parent of the current directory. -pvh -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada
Re: Reiser4 Pseudo Directory Segfault
Edward, we are running reiser4-for-2.6.12-3.patch.gz on 2.6.12 Debian. Your patch worked perfectly and directed us to some code that answered a few other questions we had. Thank you. -pvh On December 15, 2005 11:33 am, Edward Shishkin wrote: Peter van Hardenberg wrote: How to produce the error: $ chmod +w file/ $ touch file//newattr Segmentation fault. What the basic setup (kernel, patches) do you use? Indeed, in the old one the problem exists, because init_pseudo() does not care about missed regular plugin, so the attached patch fixes the problem. Thanks, Edward. We tried setting dir_eperm on the pseudo directory plugin's create member. Why didn't this fix it? -pvh -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada
Plugin structure documentation request
It would be quite excellent if we could have even a one sentence description of all the plugin structures. This structure includes many operations not found in conventional VFS, and many of the calls are not described in any documentation I have found. I must confess I am feeling some frustration at the complete lack of documentation of the project. I have very few complaints about the quality of the code, but despite having spent four months working through file after file I still regularly stumble across areas we have not had an excuse to delve into and have no idea how they work in a general sense or what they do. This kind of documentation does not take particularly long to generate but would save new users of the code many, MANY long hours spent poring over code and tracking function calls through the enormous structure. I know you have heard it from the kernel mailing list before, but now you are hearing it from someone uninvolved in that process. A trivial script reports 93866 lines of code in the reiser4 directory. That's a lot of code to navigate without a map. Having got that off my chest, here is one example of a plugin struct that could use documentation. There are several other examples -- notably the pseudo.c structure which is relatively uncomplicated and the directory plugin which is about as complicated as this example file plugin. [PSEUDO_FILE_PLUGIN_ID] = { .h = { .type_id = REISER4_FILE_PLUGIN_TYPE, .id = PSEUDO_FILE_PLUGIN_ID, .pops = file_plugin_ops, .label = pseudo, .desc = pseudo file, .linkage = TYPE_SAFE_LIST_LINK_ZERO }, .open = open_pseudo, .truncate = eperm, .write_sd_by_inode = eperm, .readpage = eperm, .capturepage = NULL, .capture = NULL, .read = read_pseudo, .write = write_pseudo, .release = release_pseudo, .ioctl = eperm, .mmap = eperm, .sync = sync_common, .get_block = eperm, .flow_by_inode = NULL, .key_by_inode = NULL, .set_plug_in_inode = set_plug_in_inode_common, .adjust_to_parent = NULL, .create= NULL, .delete= eperm, .add_link = NULL, .rem_link = NULL, .owns_item = NULL, .can_add_link = cannot, .can_rem_link = cannot, .setattr = inode_setattr, .getattr = getattr_common, .seek = seek_pseudo, .detach= detach_common, .bind = bind_common, .safelink = NULL, .estimate = { .create = NULL, .update = NULL, .unlink = NULL }, .wire = { .write = wire_write_pseudo, .read = wire_read_pseudo, .get = wire_get_pseudo, .size = wire_size_pseudo, .done = wire_done_pseudo }, .init_inode_data = NULL, .pre_delete = NULL, .cut_tree_worker = cut_tree_worker_common, .destroy_inode = NULL, }, Thanks, -pvh -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada
Reiser4 Pseudo Directory Segfault
How to produce the error: $ chmod +w file/ $ touch file//newattr Segmentation fault. We tried setting dir_eperm on the pseudo directory plugin's create member. Why didn't this fix it? -pvh -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada
Re: Two usage examples for attribute directories.
Peter van Hardenberg wrote: Still, say we'd like to be able to also specify some attributes for app, let's say author, version, name, description, contact, category -- things that would be useful for creating a menu entry. Now the directory begins to get ugly... $ ls app/ author bin category contact description lib name src var version Makefile icon.png install.sh But which are attributes? If only our namespace provided us a way to seperate one group of files from another... Aha! Why not a directory? You say author should be an attribute but icon.png is not an attribute. I am not convinced they should be treated differently. The listing doesn't look ugly to me at all (just because it is long). Ahh, much better. It also ensures we can preserve Hans' goal of not enforcing unnatural structure. What if I want to find all the files Hans is in any way associated with, whether it is as author, contact, inspiration, whatever. I don't want to look into EVERY file on the disk if I don't have to. Instead, I can just $ grep Hans Reiser */@/* I don't think this is a nice solution. The nice solution is what is suggested in Hans' whitepaper: [HansReiser] which would give you all objects associated with Hans, and this would be efficient (unlike */@/*). (Read http://www.namesys.com/whitepaper.html again.) without having to grep through gig upon gig of other data. Of course, if that's what I want to do, the option is still there... SO! Why privilege the @ directory with its own plugin? Quite. You shouldn't. Why not just use it as a convention? Efficiency. Files governed by the @ plug-in could have size restrictions, Arbitrary limits are disliked in Linux, with good reason. should be packed tightly with their parents. They might have special format to fit more into a single disk read. They could be indexed by the operating system to make searching for particular values fast. An efficient implementation of [ ] would do the indexing, or would have the same effect. Thus, I conclude that while file-directories are valuable and closely related to attribute directories, but the two serve different roles and have different design goals, so should be served by different plugins. Where does the term attribute directory come from? Did you invent it? Are they your @ directories, or is that referring to something more? Thanks, Peter
Re: some details about winfs
Is there a transcript anywhere? -pvh On December 4, 2005 10:44 pm, Bedros Hanounik wrote: an interview with MS transactional file system guy (winfs) here http://channel9.msdn.com/Showpost.aspx?postid=142120 it's in wmv format; you need mplayer with win32codecs library. It's 140MB file; and very informative, but made me think less of winFS. -B -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada
Re: Two usage examples for attribute directories.
Hi Peter F, Hans, and all, we had a great group meeting today and I think managed to come up with some reasonable explanations. I will break the threading a bit to answer the questions coherently, but I believe this answers all questions asked, so if I've missed any, please let me know. Let me begin by assuming that attributes are not in any way different from normal files and show how that leads to some limitations. I will imagine a more detailed use case. Bundled apps. I would love to be able to distribute applications as single-file bundles with all their resources in them, like this: -- $ # look at the resource files contained in app/ $ ls app/ bin lib src var Makefile icon.png install.sh $ # run app $ ./app This version of app is not built for your platform. Please run cd app/; ./configure; make; to update the build for your platform. $ # okay, we'll recompile app for our system $ cd app/; ./configure; make; Compiling app... Done. $ # let's see if it works $ ./app Starting app... $ # yep, looks good! let's install it. $ sudo app/install.sh --prefix=/usr/local Installing app into /usr/local... # Note that app is now installed so the ./ is no longer needed. $ app Starting app... $ # done. -- That demonstrated using file-directories to contain resources. Any file-dir application would add itself to its own path on startup. This would be supremely useful for projects like Klik, and implements (I would say) ideally the things that OS X is currently emulating with their prog.app directories. Still, say we'd like to be able to also specify some attributes for app, let's say author, version, name, description, contact, category -- things that would be useful for creating a menu entry. Now the directory begins to get ugly... $ ls app/ author bin category contact description lib name src var version Makefile icon.png install.sh But which are attributes? If only our namespace provided us a way to seperate one group of files from another... Aha! Why not a directory? $ ls app/@/ author category contact description name version $ ls app/ bin lib src var Makefile icon.png install.sh Ahh, much better. It also ensures we can preserve Hans' goal of not enforcing unnatural structure. What if I want to find all the files Hans is in any way associated with, whether it is as author, contact, inspiration, whatever. I don't want to look into EVERY file on the disk if I don't have to. Instead, I can just $ grep Hans Reiser */@/* without having to grep through gig upon gig of other data. Of course, if that's what I want to do, the option is still there... SO! Why privilege the @ directory with its own plugin? Why not just use it as a convention? Efficiency. Files governed by the @ plug-in could have size restrictions, and should be packed tightly with their parents. They might have special format to fit more into a single disk read. They could be indexed by the operating system to make searching for particular values fast. Control. They might have strict data types or controlled grammars. The @ directory might not actually be displayed in a standard ls of the directory (like isn't). Compatibility. The @ directory plugin should definitely have compatibility with xattr syntax so that existing software would be able to use Reiser attributes out of the box without any source changes. That would be a big win for driving adoption. Thus, I conclude that while file-directories are valuable and closely related to attribute directories, but the two serve different roles and have different design goals, so should be served by different plugins. On December 3, 2005 02:54 am, Peter Foldiak wrote: Are you sure the simple / is not a more elegant and simple way to do all this.Peter No, I'm not sure, but I'm hoping we'll find one by talking about it. -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada
Re: Two usage examples for attribute directories.
Peter van Hardenberg wrote: PvH, I am really worried about introducing this new concept of attribute directory and new syntax. Maybe I am just missing the point (in which case please explain), but is there anything you can do with this that there is no way you could do without it. Hans' favourite topic (quite rightly) is namespace unification, which involves keeping the syntax as simple as possible, having only one type of expression for one kind of logical structure. If you could possibly do this without the ..@, do it without it. So why not just do (to take your example): $ echo Matmos track01.mp3/artist Peter F This is not new syntax, it is a new type of File (well, tecnically, it would be implemented as a new pseudoplugin.) But why do we need a new type of file? What can you do with it that you absolutely couldn't without? What is wrong with my simplification of your example? (track01.mp3/artist) From a user's point of view, these files are equivalent to an XML attribute, whereas normal files are child elements. I never quite understood the need for XML attributes either. You could easily survive without them, right? The semantic difference is, I contend, significant. Could you explain this difference. From an implementation point of view, this would allow us to provide guidance to, or eventually even full FS-level support for indexed attributes. Sorry for my ignorance, what is an indexed attribute and what's good about it. Are you sure the simple / is not a more elegant and simple way to do all this.Peter
Re: Two usage examples for attribute directories.
Peter van Hardenberg wrote: On November 20, 2005 11:47 pm, Hans Reiser wrote: Peter van Hardenberg wrote: We have an implementation plan for the attribute plugin. We plan to base it around the plugin.c so that it can be available for all files, directory or otherwise. Every file which has a pseudo will gain a new user-attributes pseudofile. This pseudofile will essentially wrap a standard directory. Usage examples please. - Simple moves first: $ echo Matmos track01.mp3/..@/artist $ echo A Chance to Cut is a Chance to Cure track01.mp3/..@/album $ grep Matmos track01.mp3/..@/* artist: Matmos - Some hypothetical pseudocode from a music player: playcount = open('track01.mp3/..@/amarok/playcount').read; playcount = playcount + 1; open('track01.mp3/..@/amarok/playcount', 'w').write(playcount); - PvH, I am really worried about introducing this new concept of attribute directory and new syntax. Maybe I am just missing the point (in which case please explain), but is there anything you can do with this that there is no way you could do without it. Hans' favourite topic (quite rightly) is namespace unification, which involves keeping the syntax as simple as possible, having only one type of expression for one kind of logical structure. If you could possibly do this without the ..@, do it without it. So why not just do (to take your example): $ echo Matmos track01.mp3/artist Peter F
Re: Two usage examples for attribute directories.
PvH, I am really worried about introducing this new concept of attribute directory and new syntax. Maybe I am just missing the point (in which case please explain), but is there anything you can do with this that there is no way you could do without it. Hans' favourite topic (quite rightly) is namespace unification, which involves keeping the syntax as simple as possible, having only one type of expression for one kind of logical structure. If you could possibly do this without the ..@, do it without it. So why not just do (to take your example): $ echo Matmos track01.mp3/artist Peter F This is not new syntax, it is a new type of File (well, tecnically, it would be implemented as a new pseudoplugin.) From a user's point of view, these files are equivalent to an XML attribute, whereas normal files are child elements. The semantic difference is, I contend, significant. From an implementation point of view, this would allow us to provide guidance to, or eventually even full FS-level support for indexed attributes. These attribute directories, like would also be available for directories, not just files. Also, important to me is that they could provide Reiser4 xattr compatibility. (The xattr('artist') for a file would be file/@/artist.) Personally, I think ..@ is a keyboard-full. I would prefer @ for improved consistency with xpath, but worrying about the actual string to access with is worrying about what colour the shed will be before you put it up. I think resource directories in the style of OSX have a seperate value to them and also deserve implementation, but are another (related) project. Dinner beckons, -pvh -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada
Two usage examples for attribute directories.
On November 20, 2005 11:47 pm, Hans Reiser wrote: Peter van Hardenberg wrote: We have an implementation plan for the attribute plugin. We plan to base it around the plugin.c so that it can be available for all files, directory or otherwise. Every file which has a pseudo will gain a new user-attributes pseudofile. This pseudofile will essentially wrap a standard directory. Usage examples please. - Simple moves first: $ echo Matmos track01.mp3/..@/artist $ echo A Chance to Cut is a Chance to Cure track01.mp3/..@/album $ grep Matmos track01.mp3/..@/* artist: Matmos - Some hypothetical pseudocode from a music player: playcount = open('track01.mp3/..@/amarok/playcount').read; playcount = playcount + 1; open('track01.mp3/..@/amarok/playcount', 'w').write(playcount); - -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada
Plugin modifications to disk format.
More codework today, so more questions. inodes: -- We would like to add a new field to every inode on disc. This could monopolize the plugin_data union on the struct. Is there a better way? How can this be achieved without breaking format compatibility? Storage: --- How tightly can these attribute directories be packed? Dominic Giampaulo rejected a similar design to ours for efficiency purposes in the BeFS, but he was not lucky enough to have the advantages of the Reiser storage layer underneath. So... since we have those advantages, how tight can we pack the following: Small file with an attribute directory: inode -data -attributedir_inode -attributedir_ dir_file -attribute_inode -attribute_data -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada
Implementing an attribute directory.
We have an implementation plan for the attribute plugin. We plan to base it around the plugin.c so that it can be available for all files, directory or otherwise. Every file which has a pseudo will gain a new user-attributes pseudofile. This pseudofile will essentially wrap a standard directory. In order to provide support for this directory, this plugin will include a new field on the inode which points to a standard dir_file. When a new file is created, this inode will be created as well, and when a file is deleted, this one must be removed (with children) as well. Future versions should provide the optimization of only creating the inode once it becomes needed and otherwise simulating an empty directory. Namesys folks, does this seem like a good course of action? -pvh -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada
Re: Erratum
On November 17, 2005 08:13 pm, Leo Comerford wrote: - this is what comes of writing things in a hurry Leo, As you may have noticed, I am working in this area right now. I'd like to read what you have written, but it's too long and jargon-laden for me to invest the time. Could you sum up everything you said into about a hundred words such that an educated layman could understand? I believe Einstein is quoted as saying forgive the length of this letter, I did not have time to make it short so please don't take this as a complaint about your idea, just about its presentation. -pvh -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada
Regular Plugin
The notion of the regular plugin seems odd to me. It is a seperate plugin type with distinct plugins that only serve to refer to file plugins (which are found in plugin/object.c, not in plugin/file/*.c). So when I create a new file plugin, I cannot use it unless I also create a regular plugin which serves no purpose except to allow me to create those files by storing their ID. A few words explaining why the regular plugin field couldn't have somehow storeda file plugin would be educational. Also, regular is a rather unclear name. Something like defaultfileplugin might be easier to deduce, or even defaultchild. -pvh -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada
Re: Technical plugin details
On November 12, 2005 12:32 pm, you wrote: Peter van Hardenberg wrote: Hans, I am having trouble modifying a file to use our new plugin. I've tried writing various values into foo/.^4/plugin/file but it won't stick. This is a business of plugin-h.pops-change() method. And for regular file plugin this method (change_file) does nothing. Thanks. That was exactly what we were trying to find. We did find it eventually, after experimenting with some other plugin types to determine what the problem was. Of course, we are still trying to figure out how to actually create a file with a different file plugin. I guess I will have to read the reiser4(...) function as I can see no way to specify these things from a standard bash shell. -pvh -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada
Versioning Plugin
On November 11, 2005 05:59 am, John Gilmore wrote: Does anybody remember GoBack? It was a versioning system for windows 95/98 that was incredibly flexible and useful. Tracked all changes to the whole disk. Old versions of a file? no problem. grab an old version of a directory for referance temporarily? easy. Got a virus? revert the whole HD, and then grab the newer copies of your documents and saved games as needed. My thoughts on this: The versioning would be an audit plugin. When the file is modified, tag the current version, copy it into a sub-directory (oh, I don't know, say file/.revisions/number/date), and disable write access to it. You might not even need extended filesystem attributes for this, but they would be handy for tagging particular versions. Copy-on-write would make this action extremely cheap, only adding a couple of extra writes to make it work. Given working resource directories, COW, and the ability to set plugins, this might be a relatively easy hack to implement. Given an efficient xpath shell, you could even create a view of your drive on a particular day. If you had a file that was changing often, perhaps you could set an attribute on that file which told it only to clone the file every once in a while. Come to think of it, a userspace daemon could run in the background and replace the need for a plugin, which is probably the better solution. Then you just need COW and files which can contain resources. -pvh -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada
Technical plugin details
Hans, I am having trouble modifying a file to use our new plugin. I've tried writing various values into foo/.^4/plugin/file but it won't stick. The code is rather sparse on comments in this part of the tree, and this is really something I would hope is documented somewhere. Can you point me in the right direction? -pvh -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada
Re: Plugins: Pseudo-fun.
On Wednesday 09 November 2005 19:31, Hans Reiser wrote: Peter van Hardenberg wrote: On November 9, 2005 10:02 am, Hans Reiser wrote: Although the overall goal remains elusive, we think we should be able to implement a new Reiser4 pseudo directory in 4dot (//). This directory, called user will contain per-file user-defined attributes. Why would these go into the pseudo directory instead of into the directory? I think /user/hat-size is too long, and /hat-size is just fine if they go into . The interesting question though is should they go into or into . Alternatively, they should go into ., and that way it can be known whether they were created by users and are things that are not contained by the object but are instead meta data about the object. so, we can either call it robert/hat-size or robert//hat-size or robert/./hat-size or robert//user/hat-size, or, perhaps best of all, call it BOTH robert/./hat-size and robert/hat-size and also have a contained-by/ for distinguishing the objects that are contained in a directory rather than describe the directory, as in robert/contained-by/stomach I think I really prefer the last. Note that it requires 0 lines of code beyond allowing files to be directories. Peter Foldiak (it seems we have two Peters now so I will be forced into formality.), please comment on this, as I value your intuition in semantic matters. Thanks. I think it is most important to have naming conventions (at least on the long run) that reflect the logical structure and relationships between objects AND NOTHING ELSE. The reason for wanting nothing other than the logical structure is that everything else must be arbitrary and must be memorised. I think one of Hans' main points in his future vision white paper is that we should not have to learn the particular form of organisation that the people who stored an object decided to make up, as everything about naming should be natural and obvious from the logical structure of the objects. (Hans gives examples as how bad it is when you have to guess what fields are used in a database.) So on this list (in connection with part-whole hierarchies) I argued previously that paragraph 2 of chapter 3 of a book should be named book/chapter[3]/paragraph[2] and not anything else. It should NOT matter whether the person who scanned the book decided to put each chapter into a separate file (whatever that concept will become) or into one big file with some way (e.g. markup, e.g. XML) to indicate the structure within that file. You can read this as paragraph 2 OF chapter 3 OF the book. I think I managed to convince Hans that on the long run, it would be much better to be able to cat /home/peter/book instead of cat /home/peter/book//childcat which is neither natural or obvious. The trouble with this example is just that the person using the naming should not be required to know how the book is stored (by chapters or as one big object). [The whole point of uniting files and directories into objects is that this distinction should eventually vanish anyway. What you have on disk is a tree structure, and whether you give different parts of that tree should just be a matter of convenience. Which bits have metadata connected to them is another matter.] Similarly, I would argue that if you want to know Robert's hat size, you should use Robert/hat-size and read it as hat-size OF Robert. This would be analogous to the conventional /etc/passwd read as the passwd file OF the etc directory. Another reading of the / operator would be the value of, as in subject/strike to/elves from/Santa which would read as the subject IS strike, the sender IS Santa, ... (It remains to be seem whether the IS and OF interpretation of / clash at some point. I can't see it at the moment.) Anyway, my point is that if we can implement Robert/size for Robert's size then it is much preferable to anything more complicated-looking. Also, possibly Robert//size should refer to the size of the Robert file (or object), rather than Robert's own size. This would be a very useful distinction to be able to make. Peter (Foldiak)
Re: Plugins: Pseudo-fun.
On Thursday 10 November 2005 12:17, Peter Foldiak wrote: one big object). [The whole point of uniting files and directories into objects is that this distinction should eventually vanish anyway. What you have on disk is a tree structure, and whether you give different parts of CORRECTION. what I meant to say here was: whether you give different names to parts of ... that tree should just be a matter of convenience. Which bits have metadata connected to them is another matter.]
Re: Plugins: Pseudo-fun.
On Thursday 10 November 2005 12:26, PFC wrote: book/chapter[3]/paragraph[2] I'd rather nitpick this to be : book/chapters/3/paragraphs/2 You are probably right, your version fits the Unix syntax and Hans' syntax better. The first version was suggested only to be compatible with XPath syntax (so that you could use XPath for the whole filesystem, not just within a file [again, blurring the file boundary]). I don't know how important that would turn out to be. PF
Re: Plugins: Pseudo-fun.
On Thursday 10 November 2005 17:16, Hans Reiser wrote: cat /home/peter/book instead of cat /home/peter/book//childcat yes, but the creator of book should define that book/default is linked to /childcat, and should be able to make it something else if he chooses. Yes, it should be possible to make it something else. But perhaps if all you want is simple concatenation, that could be automatic (unless something else is specified). (And should it be book//default, after all, isn't this typical metadata?) What is the best way to specify such an object constructor? (Should it be just a script? But that wouldn't work in the kernel, right?) PF
Re: Plugins: Pseudo-fun.
On November 10, 2005 04:36 am, Peter Foldiak wrote: On Thursday 10 November 2005 12:26, PFC wrote: book/chapter[3]/paragraph[2] I'd rather nitpick this to be : book/chapters/3/paragraphs/2 You are probably right, your version fits the Unix syntax and Hans' syntax better. The first version was suggested only to be compatible with XPath syntax (so that you could use XPath for the whole filesystem, not just within a file [again, blurring the file boundary]). I don't know how important that would turn out to be. PF PF, I intend to work on a(n inefficient) proof-of-concept xPath shell once this attribute notation / storage mechanism is finished. -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada
Plugins: Pseudo-fun.
Project update: --- Well, we have wallpapered one of the labs here at UVic with Reiser4 code, printing page after page and taping them together, hanging them on the walls and highlighting them. This was the first time in years that I missed having a dot-matrix tractor-feed printer around. Tracing the execution down through the various chains of plugin methods into the plugin set and then into pseudo.c. Pseudo.c is a much better example file than dirc or file.c, the comments are superior and there are many small examples of plugins. Here we had our insights and our inspiration: Although the overall goal remains elusive, we think we should be able to implement a new Reiser4 pseudo directory in 4dot (//). This directory, called user will contain per-file user-defined attributes. These will be (if I am using the terminology correctly) grandparent-locality-packed and will be, by convention, very small. (Perhaps the user attributes should be somewhere else? ...attributes vs ...system? I know this is one of those ongoing debates.) Playing nice with xattr: It is my opinion that xattr support could be added to provide an alternate interface to the same mechanism, thus applications expecting xattr would be able to both read and operate on reiser file-style attributes, a major win for interoperability. Gotta read more. Big Question of the Day: -- One big question that follows from today's discoveries and plans: What would be the best method to store these small attribute files? I see no need to reinvent the wheel -- this is a directory -- but how/where do we store the data? Suggestions welcome. Odd Behavior: -- Regression problems immediately spring to mind: [EMAIL PROTECTED]:~/reiser/test $ cd [EMAIL PROTECTED]:~/reiser/test/ $ cd [EMAIL PROTECTED]:~/reiser/test// $ cd [EMAIL PROTECTED]:~/reiser/test/// $ cd [EMAIL PROTECTED]:~/reiser/test//// $ cd [EMAIL PROTECTED]:~/reiser/test///// $ cd [EMAIL PROTECTED]:~/reiser/test////// $ cd [EMAIL PROTECTED]:~/reiser/test/////// $ cd .. [EMAIL PROTECTED]:~/reiser/test////// $ cd .. [EMAIL PROTECTED]:~/reiser/test///// $ cd .. [EMAIL PROTECTED]:~/reiser/test//// $ cd .. [EMAIL PROTECTED]:~/reiser/test/// $ cd .. [EMAIL PROTECTED]:~/reiser/test// $ cd .. [EMAIL PROTECTED]:~/reiser/test/ $ Sure, it's right ( is of type pseudo directory), but I wouldn't want my directory tree traversal to ever end up in the bottomless pit of 4dot-doom by mistake. Crashing and glitching: [EMAIL PROTECTED]:~/reiser/ $ cd A2.mp3/...tab freezes the console. I think regular files, when pseudo data is enabled, need to provide some basic directory functionality so that users don't accidentally blow things up. [EMAIL PROTECTED]:~/reiser/A2.mp3/ $ ls ls: reading directory .: Not a directory How would you recommend we implement this so that these errors go away? Neat stuff: [EMAIL PROTECTED]:~/reiser/test/ $ ls .. A2.mp3 boo test [EMAIL PROTECTED]:~/reiser/test/ $ perl open(N, '', 'new'); print(N newfile); close(N); [EMAIL PROTECTED]:~/reiser/test/ $ ls .. A2.mp3 boo newfile test [EMAIL PROTECTED]:~/reiser/test/ $ But what's it for? I was hoping to be able to actually create new attributes on a normal file with this, but no dice. It only works for directories. That's probably more than enough for one night, but we had a real marathon session today and have another planned for tomorrow starting in the early afternoon. Hope people are finding these interesting, -pvh -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada
Re: Plugins: Pseudo-fun.
On November 9, 2005 10:02 am, Hans Reiser wrote: Although the overall goal remains elusive, we think we should be able to implement a new Reiser4 pseudo directory in 4dot (//). This directory, called user will contain per-file user-defined attributes. Why would these go into the pseudo directory instead of into the directory? Let me construct a simple use case for having a namespace separation between contained data and metadata. For example, a self-installer or a Klik style bundled package could include all its resources within its own directory. In this hypothetical use case, it would be very valuable for the package metadata (name, version, description) to be clearly differentiated from the package data. I do not have a strong opinion about whether user metadata should be a sibling of fourdot or a child. You don't want to have directory traversals use at all. No, we really don't. So far, my experiments with ls and grep show them playing nicely, but that may only be because '' doesn't get listed by 'ls -la'. Crashing and glitching: [EMAIL PROTECTED]:~/reiser/ $ cd A2.mp3/...tab freezes the console. I think regular files, when pseudo data is enabled, need to provide some basic directory functionality so that users don't accidentally blow things up. Why exactly does it freeze things? I have no idea. I suspect tab-completion in my console is waiting for a call to file-locate() to come back and doesn't know how to cope when it doesn't. I hate to say it, but you are the ones most actively working on this now, and I encourage you to fix the little bugs you find as you go, and tell us about them. Let me know about everything that affects semantics/functionality before fixing it though I'll keep sending out these emails. -pvh -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada
Re: Plugins: Pseudo-fun.
On November 9, 2005 12:38 pm, michael chang wrote: On 11/9/05, Hans Reiser [EMAIL PROTECTED] wrote: Considering everything you are working on (it's great, by the way - keep up the good work, we (the users) appreciate it!), this is probably irrelevant, but you can do what little schoolchildren do: call them Peter F and Peter H (or Peter van H), provided they agree. That should be a reasonable compromise? pvh is a fine substitute for Peter when there is ambiguity -- I've been using it long enough now that it looks like my name to me. -pvh -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada
More from the Weekday Warriors
Project News: - Alright, today was another good solid eight hour day of Reiser hacking. We finally have most of a grip on the specifics of how the pseudo-plugins work, how they interface with the filesystem, and so on. Our plan has further matured: the filedir (pronounced rather like folder) plugin will be a regular file-type plugin, and will simply inherit most of its methods from file. As the file-contents will already be monopolizing the i-node, a filedir will also have a pointer to a second directory inode which will contain the filedir's contents. All directory-related lookups will be routed through to that. The rough implementation plan is as follows: 1) Attempt a trivial change to Reiser and ensure it loads. (Done.) 2) Clone an existing plugin. (In progress.) 3) Replace existing plugin's methods with pass-through methods. 4) Associate a directory (with requisite plugin) with the filedir. Plugin Project Statistics: -- pages of code printed: 200 hours spent reading Reiser4 code: 25 lines of code modified and compiled: 1 Where should attributes live: Hans, I like the idea that files should be (for those who want it) treatable indistinguishably from directories. The contained-by would get very tedious with deep nesting, as that is already how the / translates for me. I propose this: files /are/ a normal directory (for resources) and /have/ an invisible subdirectory (for attributes) (perhaps fivedot?). The use of a seperate ontological entity for attributes is defensible: first, without it, the user attributes are visible in a way system attributes are not. More significantly, by separating the two they can be packed differently and the attributes could have special properties. Without too much thought about the consequences, I'd probably want indexing, two-way compatibility with xattr, compression, and perhaps something like size/format restrictions. I can think of about a half-dozen other attribute-specific features and optimizations I'd want in a perfect world, but I'll leave it at that for now. Question for the Day: --- What files need modification to add a new plugin which does not require inode extensions? How can I set the plugins governing a file from userspace? Can it be done from within fourdot? My experiments were unsuccessful. What would a hello world plugin look like? It would be valuable to have one lying around for future developers. We have discussed creating one, but I feel as though plugging and unplugging plugins requires a huge amount of work right now and that would reduce its utility. Perhaps it would make sense if it were distributed as a patch with a plugin developer's guide. -pvh -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada
ReiserFS problem ?
Title: ReiserFS problem ? Hi all, given situation: HP DL580 G2 / SuSE SLES8 / ReiserFS3 Last friday the server stops with the following massage: Nov 4 09:26:44 kernel: journal-1996: do_journal_end, could not get a list bitmap Nov 4 09:26:44 kernel: kernel BUG at prints.c:334! Nov 4 09:26:44 kernel: invalid operand: 2.4.21-292-smp #1 SMP Tue May 24 16:59:12 UTC 2005 Nov 4 09:26:44 kernel: CPU: 1 Nov 4 09:26:44 kernel: EIP: 0010:[ide-cd:__insmod_ide-cd_O/lib/modules/2.4.21-292-smp/kernel/drivers-55611672/96] Tainted: P Nov 4 09:26:44 kernel: EIP: 0010:[ce156ee8] Tainted: P Nov 4 09:26:44 kernel: EFLAGS: 00010296 Nov 4 09:26:44 kernel: eax: 003d ebx: cdd59800 ecx: 0046 edx: c0331968 Nov 4 09:26:44 kernel: esi: edi: ebp: de2da660 esp: f6e05e88 Nov 4 09:26:44 kernel: ds: 0018 es: 0018 ss: 0018 Nov 4 09:26:44 kernel: Process db2sysc (pid: 15060, stackpage=f6e05000) Nov 4 09:26:44 kernel: Stack: ce174e16 ce1758c0 cdd59800 ce1684ff cdd59800 ce173fc0 1000 Nov 4 09:26:44 kernel: 0006 ce162701 f6e04000 0004 0002 d2659420 0004 Nov 4 09:26:44 kernel: 0001 1a69 dadb2820 dadb2620 e1b4b000 db669000 f92e274c Nov 4 09:26:44 kernel: Call Trace: [ide-cd:__insmod_ide-cd_O/lib/modules/2.4.21-292-smp/kernel/drivers-55489002/96] (04) [ide-cd:__insmod_ide-cd_O/lib/modules/2.4.21-292-smp/kernel/drivers-55486272/96] (12) [ide-cd:__insmod_ide-cd_O/lib/modules/2.4.21-292-smp/kernel/drivers-55540481/96] (08) Nov 4 09:26:44 kernel: Call Trace: [ce174e16] (04) [ce1758c0] (12) [ce1684ff] (08) Nov 4 09:26:44 kernel: [ide-cd:__insmod_ide-cd_O/lib/modules/2.4.21-292-smp/kernel/drivers-55492672/96] (12) [ide-cd:__insmod_ide-cd_O/lib/modules/2.4.21-292-smp/kernel/drivers-55564543/96] (76) [ide-cd:__insmod_ide-cd_O/lib/modules/2.4.21-292-smp/kernel/drivers-55545414/96] (28) [ide-cd:__insmod_ide-cd_O/lib/modules/2.4.21-292-smp/kernel/drivers-55543250/96] (44) Nov 4 09:26:44 kernel: [ce173fc0] (12) [ce162701] (76) [ce1671ba] (28) [ce167a2e] (44) Nov 4 09:26:45 kernel: [generic_file_write/416] (20) [ide-cd:__insmod_ide-cd_O/lib/modules/2.4.21-292-smp/kernel/drivers-55543133/96] (24) [ide-cd:__insmod_ide-cd_O/lib/modules/2.4.21-292-smp/kernel/drivers-55643128/96] (32) [sys_pwrite/376] (52) Nov 4 09:26:45 kernel: [c014416f] (20) [ce167aa3] (24) [ce14f408] (32) [c0156e6a] (52) Nov 4 09:26:45 kernel: [system_call/56] (60) Nov 4 09:26:45 kernel: [c0109727] (60) Nov 4 09:26:45 kernel: Modules: [(reiserfs:ce140060:ce177274)] Nov 4 09:26:45 kernel: Code: 0f 0b 4e 01 1c 4e 17 ce 85 db 74 0e 0f b7 43 08 89 04 24 e8 Nov 4 09:26:45 kernel: klogd 1.4.1, -- state change -- Nov 4 09:26:45 kernel: Inspecting /boot/System.map-2.4.21-292-smp Nov 4 09:26:45 kernel: Loaded 22038 symbols from /boot/System.map-2.4.21-292-smp. Nov 4 09:26:45 kernel: Symbols match kernel version 2.4.21. Nov 4 09:26:45 kernel: Loaded 487 symbols from 13 modules. An reboot of the machine stops with: kernel BUG at journal.c:782! invalid operand: 000 2.4.21-292-smp #1 SMP Tue May 24 16:59:12 UTC 2005 CPU: 2 EIP: 0010:[ce122cec] Not tainted EEFLAGS:00010246 eax: ebx: ecx: ce15cf24 edx: f92820d8 esi: ce15cee0 edi: ebp: d0958c00 esp: cded1e98 ds: 0018 es: 0018 ss: 0018 Prozess init (pid: 1, stackpage=cded1000) Stack: d0958c00 c0159078 0286 1a70 f9282078 ce15cef8 0286 f92820f0 ce15cee0 ce1284e6 d0958c00 ce15cee0 0001 0006 ce126635 cded 0002 Call Trace: [c0159078] (48) [ce1284e6] (20) [ce126635] (72) [ce1137a0] (04) [ce1271ba] (28) [ce11383c] (60) [c015d826] (28) [c01582a7] (20) [c01583cf] (08) [c0109727] (60) Modules: [(reiserfs:ce1583cf:ce137274)] Code: 0f 0b 0e 03 59 50 13 ce 83 7e 64 01 0f 8e 19 04 00 00 8b 46 0Kernel panic: Attempted to kill init! Any suggestions? Need more Infos? Thanks.
kernel BUG at prints.c:334!
Hi all, SORRY for the first html mail! given situation: HP DL580 G2 / SuSE SLES8 / ReiserFS3 Last friday the server stops with the following massage: Nov 4 09:26:44 kernel: journal-1996: do_journal_end, could not get a list bitmap Nov 4 09:26:44 kernel: kernel BUG at prints.c:334! Nov 4 09:26:44 kernel: invalid operand: 2.4.21-292-smp #1 SMP Tue May 24 16:59:12 UTC 2005 Nov 4 09:26:44 kernel: CPU:1 Nov 4 09:26:44 kernel: EIP: 0010:[ide-cd:__insmod_ide-cd_O/lib/modules/2.4.21-292-smp/kernel/drivers +-55611672/96]Tainted: P Nov 4 09:26:44 kernel: EIP:0010:[ce156ee8]Tainted: P Nov 4 09:26:44 kernel: EFLAGS: 00010296 Nov 4 09:26:44 kernel: eax: 003d ebx: cdd59800 ecx: 0046 edx: c0331968 Nov 4 09:26:44 kernel: esi: edi: ebp: de2da660 esp: f6e05e88 Nov 4 09:26:44 kernel: ds: 0018 es: 0018 ss: 0018 Nov 4 09:26:44 kernel: Process db2sysc (pid: 15060, stackpage=f6e05000) Nov 4 09:26:44 kernel: Stack: ce174e16 ce1758c0 cdd59800 ce1684ff cdd59800 ce173fc0 1000 Nov 4 09:26:44 kernel:0006 ce162701 f6e04000 0004 0002 d2659420 0004 Nov 4 09:26:44 kernel: 0001 1a69 dadb2820 dadb2620 e1b4b000 db669000 f92e274c Nov 4 09:26:44 kernel: Call Trace: [ide-cd:__insmod_ide-cd_O/lib/modules/2.4.21-292-smp/kernel/drivers+-554 89002/96] (04) [ide-cd:__insmod_ide-cd_O/lib/modules/2.4.21-292-smp/kernel/drivers+-554 86272/96] (12) [ide-cd:__insmod_ide-cd_O/lib/modules/2.4.21-292-smp/kernel/drivers+-555 40481/96] (08) Nov 4 09:26:44 kernel: Call Trace: [ce174e16] (04) [ce1758c0] (12) [ce1684ff] (08) Nov 4 09:26:44 kernel: [ide-cd:__insmod_ide-cd_O/lib/modules/2.4.21-292-smp/kernel/drivers+-554 92672/96] (12) [ide-cd:__insmod_ide-cd_O/lib/modules/2.4.21-292-smp/kernel/drivers+-555 64543/96] (76) [ide-cd:__insmod_ide-cd_O/lib/modules/2.4.21-292-smp/kernel/drivers+-555 45414/96] (28) [ide-cd:__insmod_ide-cd_O/lib/modules/2.4.21-292-smp/kernel/drivers+-555 43250/96] (44) Nov 4 09:26:44 kernel: [ce173fc0] (12) [ce162701] (76) [ce1671ba] (28) [ce167a2e] (44) Nov 4 09:26:45 kernel: [generic_file_write+383/416] (20) [ide-cd:__insmod_ide-cd_O/lib/modules/2.4.21-292-smp/kernel/drivers+-555 43133/96] (24) [ide-cd:__insmod_ide-cd_O/lib/modules/2.4.21-292-smp/kernel/drivers+-556 43128/96] (32) [sys_pwrite+202/376] (52) Nov 4 09:26:45 kernel: [c014416f] (20) [ce167aa3] (24) [ce14f408] (32) [c0156e6a] (52) Nov 4 09:26:45 kernel: [system_call+51/56] (60) Nov 4 09:26:45 kernel: [c0109727] (60) Nov 4 09:26:45 kernel: Modules: [(reiserfs:ce140060:ce177274)] Nov 4 09:26:45 kernel: Code: 0f 0b 4e 01 1c 4e 17 ce 85 db 74 0e 0f b7 43 08 89 04 24 e8 Nov 4 09:26:45 kernel: klogd 1.4.1, -- state change -- Nov 4 09:26:45 kernel: Inspecting /boot/System.map-2.4.21-292-smp Nov 4 09:26:45 kernel: Loaded 22038 symbols from /boot/System.map-2.4.21-292-smp. Nov 4 09:26:45 kernel: Symbols match kernel version 2.4.21. Nov 4 09:26:45 kernel: Loaded 487 symbols from 13 modules. An reboot of the machine stops with: kernel BUG at journal.c:782! invalid operand: 000 2.4.21-292-smp #1 SMP Tue May 24 16:59:12 UTC 2005 CPU:2 EIP:0010:[ce122cec] Not tainted EEFLAGS:00010246 eax: ebx: ecx: ce15cf24 edx: f92820d8 esi: ce15cee0 edi: ebp: d0958c00 esp: cded1e98 ds: 0018 es: 0018 ss: 0018 Prozess init (pid: 1, stackpage=cded1000) Stack: d0958c00 c0159078 0286 1a70 f9282078 ce15cef8 0286 f92820f0 ce15cee0 ce1284e6 d0958c00 ce15cee0 0001 0006 ce126635 cded 0002 Call Trace: [c0159078] (48) [ce1284e6] (20) [ce126635] (72) [ce1137a0] (04) [ce1271ba] (28) [ce11383c] (60) [c015d826] (28) [c01582a7] (20) [c01583cf] (08) [c0109727] (60) Modules: [(reiserfs:ce1583cf:ce137274)] Code: 0f 0b 0e 03 59 50 13 ce 83 7e 64 01 0f 8e 19 04 00 00 8b 46 0Kernel panic: Attempted to kill init! Any suggestions? Need more Infos? Thanks.
Plugin authoring.
Hans, We would like to spend some time and arrange a plugin that would allow all files to be usable as directories for arbitrary data. Our reading suggests that this could probably be done with an object plugin that combines the functionality of Files and Directories. As for the problem of hard links, we will simply disallow them. Although they have their uses, file-directories have their own interesting applications which I believe deserve exploration regardless of the outstanding problem. Now that we have found our course, we would appreciate some advice on getting started writing a plugin. I have read commentary that there is no plugin author's document and that the source code will reveal all to those willing to read it. Well, we are willing to read and hack, but need a few stars to navigate by. What code should we read in particular? Are there relatively simple examples we may follow? Is there API documentation for the storage layer? If weekend warriors are to be able to hack on Reiser, perhaps they can learn from our experience. According to my limited research, there are no third-party plugins yet that we can learn from. Peter -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada
Re: Our introduction to Reiser-list
On October 26, 2005 10:02 am, John Gilmore wrote: Actually, when I had first read about file-as-directory, I had assumed that the content was either dynamically generated from the on-disk metadata (like uid, gid, etc) or stored as a sideband type stream in the file itself, like some of the MAC OS file systems (and others) do, or generated via a plugin connecting to user-space, for ID3 tags on mp3 files and other information which can easily be obtained from the file itself. And I thought the whole idea was to unify the namespace and make things like ID3 tags obsolete... -pvh -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada
Re: Our introduction to Reiser-list
On October 27, 2005 04:17 am, David Masover wrote: Peter van Hardenberg wrote: On October 26, 2005 10:02 am, John Gilmore wrote: And I thought the whole idea was to unify the namespace and make things like ID3 tags obsolete... The two are not mutually exclusive. You unify the namespace, and use that to access things like ID3 tags. Of course, eventually ID3 tags become obsolete, and the information is instead stored outside of the file itself, as a separate stream (treated as a file). You'd have a standard way of serializing any given file and all its metadata, so that something like id3 doesn't have to be re-invented for every file type that has metadata, and so that similar metadata can be accessed through a standard mechanism -- searching for a particular artist should return songs (using id3 tags) and music videos (using the mpeg equivalent) and maybe even song lyrics (using separate metadata). It's much easier, more extensible, and more secure to create a utility which ties together a number of userspace metadata libraries to create static files than to move them all down into kernel space. I feel plugins providing pseudofiles should only be used when there is no viable alternative. -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada
Re: Our introduction to Reiser-list
Hubert Chan wrote: On Tue, 25 Oct 2005 17:04:55 -0700, Peter van Hardenberg [EMAIL PROTECTED] said: I envision creating a simple shell in a scripting language which will operate on an XML representation of the filesystem which maps, in the simple case, directories and files such that simple xpath queries appear identical to normal filesystem queries. Although it's not exactly the same, you may be interested in a research proposal that I wrote for a course last year on embedding XML databases into the filesystem namespace. At least some of the references may be useful. http://www.cs.uwaterloo.ca/~hy3chan/cs741project.ps Also take a look at the part of a (record-breakingly long?) thread file as a directory on this list (also copied to lkml) near and after: http://www.ussg.iu.edu/hypermail/linux/kernel/0411.3/0044.html There, I suggested that file selection should be unified with part-of-file selection using XPath-like syntax. To do what you are suggesting efficiently would be really nice. To do it inefficiently (with a shell script) may still be interesting, possibly even useful. But as XPath was originally inteded for selection inside XML files, it would also be nice if (at least for your XML files) you could select inside your files using a unified syntax! Peter
Re: Our introduction to Reiser-list
On October 26, 2005 05:44 am, you wrote: Also take a look at the part of a (record-breakingly long?) thread file as a directory on this list (also copied to lkml) near and after: http://www.ussg.iu.edu/hypermail/linux/kernel/0411.3/0044.html There, I suggested that file selection should be unified with part-of-file selection using XPath-like syntax. To do what you are suggesting efficiently would be really nice. To do it inefficiently (with a shell script) may still be interesting, possibly even useful. But as XPath was originally inteded for selection inside XML files, it would also be nice if (at least for your XML files) you could select inside your files using a unified syntax! Peter The objections raised about this on the LKML are quite valid. I could see there being value in this kind of access, and an XML plugin or a dictionary file plugin could be useful, given sufficient time to mature and address issues of stability and security. Personally, I LIKE the bytestream. I think it is a sensible enough building-block for abstraction. I imagine the filesystem as a tree with named branches that has streams hanging off it like Christmas ornaments. Ontologically, this is a nice simplification. Every node has the potential to be a file, no node has the requirement of being a file. The names are managed independently from the streams. Further, keeping the contents of streams opaque in the general case makes sense to me. Streams are already both simple and flexible, and a mechanism already exists for putting assigning a unique namespace to data. In fact, instead of creating XML files with a plugin, I would personally try splitting them into many small files and write a userspace DOM-library that maps calls like node.getChildren to calls like readdir(NODE). So: instead of XMLfile/entry/@attribute parsing an XML file, there would actually be files in there. Although I freely acknowledge my inexperience, I believe the real problems are related to graph traversal algorithms. Linus has commented on the obvious hardlink issues. I imagine there are more gremlins lurking in the shadows on this one. Garbage collectors have largely given up on reference counting, a luxury afforded by blazingly fast access to small amounts of storage. I am not particularly up on the research though. Also: I still have not been able to USE files as directories. Yes, I can reach file//, but that is only one special case. I can crash things like it was going out of style by playing with these file-directories. Does nobody have any experience with this? What kind of work is it going to take me to get this going? -pvh -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada
Our introduction to Reiser-list
Hello all, I'm a student at the University of Victoria. Between myself and a few fellow students we have embarked on a quest to do some experiments with the Reiser4 metadata system to show it off and provide some real world use cases. We'll be spending lots of time on this project between now and the end of the year. If list members have ideas for interesting experiments, please, join in and suggest them. So far, we have a few nifty ideas: 1) tie the 'extract' utility to a shell script to copy existing metadata down into the filesystem 2) create a (very slow) shell which operates on the filesystem by xpath query. doing it right would be difficult, but faking it should be easy. We are all inexperienced with kernel hacking and will probably ask some stupid questions as we go forward. Our individual interests vary and we may find that we have bitten off more than we can chew, but I expect it will be an interesting ride, if nothing else. Our first contribution will be a practical guide to installing Reiser4 (with metadata enabled) under Ubuntu 5.10. All the best, -pvh -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada
Usable metadata
Is there a means by which I can actually use files as directories? I have enabled the directories, but, well, the log speaks for itself: [EMAIL PROTECTED]:~/reiser $ cp ../artist A2.mp3/ Segmentation fault [EMAIL PROTECTED]:~/reiser $ cp ../artist A2.mp3/TAB freeze terminal . [EMAIL PROTECTED]:~/reiser $ cd A.mp3 [EMAIL PROTECTED]:~/reiser/A.mp3 $ ls ls: reading directory .: Not a directory [EMAIL PROTECTED]:~/reiser/A.mp3 $ I've hit a dead end, and I haven't found any documentation that can help yet. -pvh -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada
Re: Our introduction to Reiser-list
On October 25, 2005 04:08 pm, you wrote: Peter van Hardenberg wrote: Hello all, I'm a student at the University of Victoria. Between myself and a few fellow students we have embarked on a quest to do some experiments with the Reiser4 metadata system to show it off and provide some real world use cases. We'll be spending lots of time on this project between now and the end of the year. If list members have ideas for interesting experiments, please, join in and suggest them. So far, we have a few nifty ideas: 1) tie the 'extract' utility to a shell script to copy existing metadata down into the filesystem what is the extract utility? Extract is a utility that recognizes many different file formats and extracts their metadata. For example: [EMAIL PROTECTED]:~ $ /usr/bin/extract music/Audioslave\ -\ Like\ a\ Stone.mp3 format - 192 bps, 44100 hz, 4m58 stereo resource-type - MPEG V1 mimetype - audio/mpeg description - Audioslave: Like a Stone (final mixdown) comment - finalmix by Francis/AndyWarh genre - Rock date - 2002 album - final mixdown artist - Audioslave title - Like a Stone genre - 17 comment - date - 2002C album - final mixdownT artist - AudioslaveT title - Like a StoneT [EMAIL PROTECTED]:~ $ 2) create a (very slow) shell which operates on the filesystem by xpath query. can you say three paragraphs here? I envision creating a simple shell in a scripting language which will operate on an XML representation of the filesystem which maps, in the simple case, directories and files such that simple xpath queries appear identical to normal filesystem queries. For example, if the XML looked like this: home pvh music audioslave.mp3 artist=Audioslave/ radiohead.mp3 artist=Radiohead/ /music /pvh norbs music chuckberry.mp3 artist=Chuck Berry/ ledzeppelin.mp3 artist=Led Zeppelin/ /music /norbs /home You could retrieve the first music node via /home/pvh/music. The query //music would return a list containing the nodes in any music element. The query //norbs[//@artist = 'Chuck Berry'] would only return norbs' node if somewhere in that node was a file with a Chuck Berry artist attribute. For more information, check out any good xPath tutorial. There would have to be some minor modifications due to namespace limitations of XML elements, but the principle would remain the same. Our first contribution will be a practical guide to installing Reiser4 (with metadata enabled) under Ubuntu 5.10. Oh, that sounds great, it will go onto our website under the install button if you are kind enough to allow that. Of course. -p -- Peter van Hardenberg ([EMAIL PROTECTED]) Victoria, BC, Canada
corrupted disk
Hi List I have an interesting problem at a customer which I hope someone can shed some light on. The server is an IBM server with an multipath SCSI controller connected to a SAN with multiple 2 TB disks configured. The Operating System is SLES 8. Among other things the server runs IBM DB2. A previous contractor recommended to that the filesystems be directly created on the disk devices, NOT on disk partitions so the filesystem in question is on /dev/sdc At 06:15 this morning the following errors showed up in /var/log/messages Oct 11 06:15:03 DB2MUHASEBE kernel: journal-2332: Trying to log block 359, which is a log block Oct 11 06:15:03 DB2MUHASEBE kernel: kernel BUG at prints.c:334! Oct 11 06:15:03 DB2MUHASEBE kernel: invalid operand: 2.4.21-138-smp #1 SMP Fri Oct 31 00:51:31 UTC 2003 Oct 11 06:15:03 DB2MUHASEBE kernel: CPU:1 Oct 11 06:15:03 DB2MUHASEBE kernel: EIP:0010: [st:__insmod_st_O/lib/modules/2.4.21-138-smp/kernel/drivers/scs+-317460792/96] Tainted: P Oct 11 06:15:03 DB2MUHASEBE kernel: EIP:0010:[c5096ec8]Tainted: P Oct 11 06:15:03 DB2MUHASEBE kernel: EFLAGS: 00010296 Oct 11 06:15:03 DB2MUHASEBE kernel: eax: 003f ebx: f11ea000 ecx: 0046 edx: c032f8c8 Oct 11 06:15:03 DB2MUHASEBE kernel: esi: f9596578 edi: 0167 ebp: f958a7a0 esp: ebfe9ea4 Oct 11 06:15:03 DB2MUHASEBE kernel: ds: 0018 es: 0018 ss: 0018 Oct 11 06:15:03 DB2MUHASEBE kernel: Process db2sysc (pid: 18866, stackpage=ebfe9000) Oct 11 06:15:03 DB2MUHASEBE kernel: Stack: c50b4d56 c50b5800 f9596578 2012 c50a8758 f11ea000 c50b3fe0 0167 Oct 11 06:15:03 DB2MUHASEBE kernel:0006 c50a65a5 c50b5051 03882f46 ef8d48d8 d7c8e7a0 0004 Oct 11 06:15:03 DB2MUHASEBE kernel: 0042 e68797e0 e6879260 f6277000 f475b000 f9596578 Oct 11 06:15:03 DB2MUHASEBE kernel: Call Trace: [st:__insmod_st_O/lib/modules/2.4.21-138-smp/kernel/drivers/scs+-317338282/96] (04) [st:__insmod_st_O/ lib/modules/2.4.21-138-smp/kernel/drivers/scs+-317335552/96] (12) [st:__insmod_st_O/lib/modules/2.4.21-138-smp/kernel/drivers/scs+-317388968/96] (08) Oct 11 06:15:03 DB2MUHASEBE kernel: Call Trace: [c50b4d56] (04) [c50b5800] (12) [c50a8758] (08) Oct 11 06:15:03 DB2MUHASEBE kernel: [st:__insmod_st_O/lib/modules/2.4.21-138-smp/kernel/drivers/scs+-317341728/96] (12) [st:__insmod_st_O/lib/modules/2.4.21 -138-smp/kernel/drivers/scs+-317397595/96] (04) [st:__insmod_st_O/lib/modules/2.4.21-138-smp/kernel/drivers/scs+-317337519/96] (72) [st:__insmod_st_O/lib/modu les/2.4.21-138-smp/kernel/drivers/scs+-317394646/96] (28) Oct 11 06:15:03 DB2MUHASEBE kernel: [c50b3fe0] (12) [c50a65a5] (04) [c50b5051] (72) [c50a712a] (28) Oct 11 06:15:03 DB2MUHASEBE kernel: [st:__insmod_st_O/lib/modules/2.4.21-138-smp/kernel/drivers/scs+-317392482/96] (64) [st:__insmod_st_O/lib/modules/2.4.21 -138-smp/kernel/drivers/scs+-317392365/96] (24) [st:__insmod_st_O/lib/modules/2.4.21-138-smp/kernel/drivers/scs+-317492411/96] (20) [sys_fsync+152/208] (36) Oct 11 06:15:03 DB2MUHASEBE kernel: [c50a799e] (64) [c50a7a13] (24) [c508f345] (20) [c0157688] (36) Oct 11 06:15:03 DB2MUHASEBE kernel: [system_call+51/56] (60) Oct 11 06:15:03 DB2MUHASEBE kernel: [c01096b7] (60) Oct 11 06:15:03 DB2MUHASEBE kernel: Modules: [(reiserfs:c5080060:c50b71b4)] Oct 11 06:15:03 DB2MUHASEBE kernel: Code: 0f 0b 4e 01 5c 4d 0b c5 85 db 74 0e 0f b7 43 08 89 04 24 e8 Dmesg shows things like: sh-2021: reiserfs_read_super: can not find reiserfs on sd(8,32) FAT: bogus logical sector size 0 VFS: Can't find a valid FAT filesystem on dev 08:20. sh-2021: reiserfs_read_super: can not find reiserfs on sd(8,32) sh-2021: reiserfs_read_super: can not find reiserfs on sd(8,32) And mount now shows: mount: wrong fs type, bad option, bad superblock on /dev/sdc, or too many mounted file systems (aren't you trying to mount an extended partition, instead of some logical partition inside?) I am now doing a dd_rescue copy of the disk to another disk in the SAN as a backup which looks like it is going to take another 20 hours so in the meantime I was hoping someone might have some ideas what caused it, and the best way to recover this partition. Any ideas? -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc
Re: corrupted disk
Additional HW info: IBM X335 Server with HP2214 HBA( OEM qlogic 2300 fiber channnel card) SAN: IBM FAStT 600 Storage Device -Peter On Tuesday 11 October 2005 16:02, Peter Nixon wrote: Hi List I have an interesting problem at a customer which I hope someone can shed some light on. The server is an IBM server with an multipath SCSI controller connected to a SAN with multiple 2 TB disks configured. The Operating System is SLES 8. Among other things the server runs IBM DB2. A previous contractor recommended to that the filesystems be directly created on the disk devices, NOT on disk partitions so the filesystem in question is on /dev/sdc At 06:15 this morning the following errors showed up in /var/log/messages Oct 11 06:15:03 DB2MUHASEBE kernel: journal-2332: Trying to log block 359, which is a log block Oct 11 06:15:03 DB2MUHASEBE kernel: kernel BUG at prints.c:334! Oct 11 06:15:03 DB2MUHASEBE kernel: invalid operand: 2.4.21-138-smp #1 SMP Fri Oct 31 00:51:31 UTC 2003 Oct 11 06:15:03 DB2MUHASEBE kernel: CPU:1 Oct 11 06:15:03 DB2MUHASEBE kernel: EIP:0010: [st:__insmod_st_O/lib/modules/2.4.21-138-smp/kernel/drivers/scs+-317460792/ 96] Tainted: P Oct 11 06:15:03 DB2MUHASEBE kernel: EIP:0010:[c5096ec8]Tainted: P Oct 11 06:15:03 DB2MUHASEBE kernel: EFLAGS: 00010296 Oct 11 06:15:03 DB2MUHASEBE kernel: eax: 003f ebx: f11ea000 ecx: 0046 edx: c032f8c8 Oct 11 06:15:03 DB2MUHASEBE kernel: esi: f9596578 edi: 0167 ebp: f958a7a0 esp: ebfe9ea4 Oct 11 06:15:03 DB2MUHASEBE kernel: ds: 0018 es: 0018 ss: 0018 Oct 11 06:15:03 DB2MUHASEBE kernel: Process db2sysc (pid: 18866, stackpage=ebfe9000) Oct 11 06:15:03 DB2MUHASEBE kernel: Stack: c50b4d56 c50b5800 f9596578 2012 c50a8758 f11ea000 c50b3fe0 0167 Oct 11 06:15:03 DB2MUHASEBE kernel:0006 c50a65a5 c50b5051 03882f46 ef8d48d8 d7c8e7a0 0004 Oct 11 06:15:03 DB2MUHASEBE kernel: 0042 e68797e0 e6879260 f6277000 f475b000 f9596578 Oct 11 06:15:03 DB2MUHASEBE kernel: Call Trace: [st:__insmod_st_O/lib/modules/2.4.21-138-smp/kernel/drivers/scs+-317338282/ 96] (04) [st:__insmod_st_O/ lib/modules/2.4.21-138-smp/kernel/drivers/scs+-317335552/96] (12) [st:__insmod_st_O/lib/modules/2.4.21-138-smp/kernel/drivers/scs+-317388968/ 96] (08) Oct 11 06:15:03 DB2MUHASEBE kernel: Call Trace: [c50b4d56] (04) [c50b5800] (12) [c50a8758] (08) Oct 11 06:15:03 DB2MUHASEBE kernel: [st:__insmod_st_O/lib/modules/2.4.21-138-smp/kernel/drivers/scs+-317341728/ 96] (12) [st:__insmod_st_O/lib/modules/2.4.21 -138-smp/kernel/drivers/scs+-317397595/96] (04) [st:__insmod_st_O/lib/modules/2.4.21-138-smp/kernel/drivers/scs+-317337519/ 96] (72) [st:__insmod_st_O/lib/modu les/2.4.21-138-smp/kernel/drivers/scs+-317394646/96] (28) Oct 11 06:15:03 DB2MUHASEBE kernel: [c50b3fe0] (12) [c50a65a5] (04) [c50b5051] (72) [c50a712a] (28) Oct 11 06:15:03 DB2MUHASEBE kernel: [st:__insmod_st_O/lib/modules/2.4.21-138-smp/kernel/drivers/scs+-317392482/ 96] (64) [st:__insmod_st_O/lib/modules/2.4.21 -138-smp/kernel/drivers/scs+-317392365/96] (24) [st:__insmod_st_O/lib/modules/2.4.21-138-smp/kernel/drivers/scs+-317492411/ 96] (20) [sys_fsync+152/208] (36) Oct 11 06:15:03 DB2MUHASEBE kernel: [c50a799e] (64) [c50a7a13] (24) [c508f345] (20) [c0157688] (36) Oct 11 06:15:03 DB2MUHASEBE kernel: [system_call+51/56] (60) Oct 11 06:15:03 DB2MUHASEBE kernel: [c01096b7] (60) Oct 11 06:15:03 DB2MUHASEBE kernel: Modules: [(reiserfs:c5080060:c50b71b4)] Oct 11 06:15:03 DB2MUHASEBE kernel: Code: 0f 0b 4e 01 5c 4d 0b c5 85 db 74 0e 0f b7 43 08 89 04 24 e8 Dmesg shows things like: sh-2021: reiserfs_read_super: can not find reiserfs on sd(8,32) FAT: bogus logical sector size 0 VFS: Can't find a valid FAT filesystem on dev 08:20. sh-2021: reiserfs_read_super: can not find reiserfs on sd(8,32) sh-2021: reiserfs_read_super: can not find reiserfs on sd(8,32) And mount now shows: mount: wrong fs type, bad option, bad superblock on /dev/sdc, or too many mounted file systems (aren't you trying to mount an extended partition, instead of some logical partition inside?) I am now doing a dd_rescue copy of the disk to another disk in the SAN as a backup which looks like it is going to take another 20 hours so in the meantime I was hoping someone might have some ideas what caused it, and the best way to recover this partition. Any ideas? -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc
Re: corrupted disk
On Tuesday 11 October 2005 16:31, Sander wrote: Peter Nixon wrote (ao): At 06:15 this morning the following errors showed up in /var/log/messages Oct 11 06:15:03 DB2MUHASEBE kernel: kernel BUG at prints.c:334! Oct 11 06:15:03 DB2MUHASEBE kernel: invalid operand: 2.4.21-138-smp #1 SMP Fri Oct 31 00:51:31 UTC 2003 Oct 11 06:15:03 DB2MUHASEBE kernel: EIP:0010:[c5096ec8]Tainted: P Your kernel is very, very old and tainted. Yes. I am aware of that. As I mentioned the server is an IBM server running SUSE Linux Enterprise 8 and DB2. At the time of deployment of the server SLES 9 was not yet certified to run with DB2. Regards -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc
Re: corrupted disk
On Tuesday 11 October 2005 16:55, Sander wrote: Peter Nixon wrote (ao): On Tuesday 11 October 2005 16:31, Sander wrote: Peter Nixon wrote (ao): At 06:15 this morning the following errors showed up in /var/log/messages Oct 11 06:15:03 DB2MUHASEBE kernel: kernel BUG at prints.c:334! Oct 11 06:15:03 DB2MUHASEBE kernel: invalid operand: 2.4.21-138-smp #1 SMP Fri Oct 31 00:51:31 UTC 2003 Oct 11 06:15:03 DB2MUHASEBE kernel: EIP:0010:[c5096ec8] Tainted: P Your kernel is very, very old and tainted. Yes. I am aware of that. As I mentioned the server is an IBM server running SUSE Linux Enterprise 8 and DB2. At the time of deployment of the server SLES 9 was not yet certified to run with DB2. What I'm trying to say is that you are very unlikely to receive support on such an old kernel. And most likely the bug is fixed in a younger kernel. And, you are also running a tainted kernel. You are less likely to receive support on a tainted kernel. Yes I understand the issue, however without running an official SLES kernel the customer cannot receive support on DB2 or on their IBM hardware. If there is a bug in the kernel that causes this then I need to get SUSE to roll a new kernel and get it certified by IBM. If it is a bug in the SCSI driver (the module that taints the kernel) then I need to report it to IBM and get them to supply a fixed module. -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc
Re: corrupted disk
On Tuesday 11 October 2005 18:05, Vladimir V. Saveliev wrote: Peter Nixon wrote: Here you go. ok, I have to spend some time to decode this. As you are doing dd_rescue anyway - we can continue tomorrow, ok? Sure. I expect the dd_rescue to finish at around 9am UTC tomorrow at which point I will have a 2TB fileimage to play with. Cheers -- Peter Nixon http://www.peternixon.net/ PGP Key: http://www.peternixon.net/public.asc
Re: Interview with Hans on KernelTrap
Hans Reiser wrote: The user (or tools) don't even have to know which is the real file, which are parts of real files and which are concatenations of real files. This is really good for namespace unification across part-whole hierarchies. If you are required to use // in the middle, it breaks this I think. Could we get rid of requiring in the middle? (Would that break too much?)Peter In this specific instance, yes, but other pseudofiles would still need it. Sure, you can have only one default behaviour. (For instance, you would also need some metafiles to specify the details of how exactly the concatenation should be done, (e.g. what ordering (or structure) to use, whether to put something in between the files, etc.) in addition to all the other ones.)Peter
Re: Interview with Hans on KernelTrap
Peter Foldiak wrote: The convention (I think also approved(suggested?) by Linus at some point)... P.S. I found the reference, I was referring here to http://www.ussg.iu.edu/hypermail/linux/kernel/0408.3/0537.html By cool, I mean more than just elegant, though elegant is often the sign of really useful. And obviously I don't see all possible (good and bad) consequences.Peter
Re: journal size reiserfs vs reiser4
reiser4 reserves 5% of disk space for its internal needs. 5% of today's big disks seems a little excessive. Does reiser4 really need that much space or would less also suffice without compromising performance? Is there research available which makes up the basis for the 5% number? Thanx... ps
Re: journal size reiserfs vs reiser4
Hans Reiser wrote: Research for filesystems generally says that as you get more than 85% full the performance goes down, by a lot as you get close to 100%. 5% is probably too little rather than too much. Wow. What is all that space used for? Other journalling file systems that I have seen have limited things like journals to a much smaller space, expressed in megabytes, and a much smaller number than thought originally. The bigger journals just didn't end up adding to the performance measurably and were just considered to be a waste of space that could be used more usefully. Thanx... ps
Re: journal size reiserfs vs reiser4
Hans Reiser wrote: Wow. What is all that space used for? Other journalling file systems that I have seen have limited things like journals to a much smaller space, BSD FFS has a 10% limit unless you are root. They are correct to do so. Yes, they reserve that space so that their algorithms to choose blocks from the various cylinder groups can continue to choose blocks in decent locations. BSD FFS doesn't journal though. The SunOS UFS does journal metadata and retains the same 10% limit. It was discovered that the journal didn't need to be very big in order to maximize performance though. A very small log would do as well as a very large log. It would be interesting to pick a benchmark which represents some market that usually deploys reiersfs and then experiment with various aspects of the file system, including the reserve space. It is easy to do, mostly just time consuming and requires a fair amount of resources. ps
Re: journal size reiserfs vs reiser4
Hans Reiser wrote: Journaling, and reserving space for good allocation, are totally different concerns, I don't understand why you guys are conflating them. Agreed, that journaling and reserving space for good allocation are separate, except that often the space for the journal is reserved. Only being able to see one measurement that indicates space reserved from the file system makes it confusing. Thanx... ps
Re: reiser4progs 1.0.5 configure script and libaal 1.0.5
Hello The lastest reiser4progs (1.0.5) can't find libaal in configure script. More precisely it doesn't find libaal because of version problem Maybe it is a permission problem caused by SELinux. I had similar troubles myself. Disabling SELinux in /etc/selinux/config helped. Take a look at /var/log/messages if there are any SELinux related access-denied messages. On my system it seemed that ldconfig was not able to generate symbolic links for libaal and therefore reiser4progs were not able to find the required libaal version. Here are two mailing list entries dealing with ldconfig/SELinux problems: http://lists.freshrpms.net/pipermail/freshrpms-list/2004-December/011960.html https://www.redhat.com/archives/fedora-test-list/2004-December/msg00275.html Best regards, Peter.