Re: Is as_ops.c releasepage patch still needed?
Hi, On 21 August 2006 22:49, Jonathan Briggs wrote: The following patch was posted to the Reiser4 list August 3 by zam. Is it still needed? It solved many problems for me, making my systems able to actually complete full Beagle indexing. the patch submitted already and will be included into the next -mm kernel (after 2.6.18-rc4-mm3). But I have not seen this patch show up in the last two mm kernel releases. Did something else fix it or is this patch still needed? Index: linux-2.6-git/fs/reiser4/as_ops.c === --- linux-2.6-git.orig/fs/reiser4/as_ops.c +++ linux-2.6-git/fs/reiser4/as_ops.c @@ -350,6 +350,11 @@ int reiser4_releasepage(struct page *pag if (PageDirty(page)) return 0; + /* extra page reference is used by reiser4 to protect +* jnode-page link from this -releasepage(). */ + if (page_count(page) 3) + return 0; + /* releasable() needs jnode lock, because it looks at the jnode fields * and we need jload_lock here to avoid races with jload(). */ spin_lock_jnode(node); -- Alex.
Re: Possible bug with FIBMAP
Le lundi 28 août 2006 10:33, Alexander Zarochentsev a écrit : Reiser4: restore FIBMAP ioctl support for packed files restore FIBMAP ioctl support for packed files, don't report block numbers for not yet mapped to disk nodes. Thanks, I really couldn't fix this myself. Keep up the good work ! Brice
Re: Reiser4 und LZO compression
Nigel Cunningham wrote: Hi. On Tue, 2006-08-29 at 06:05 +0200, Jan Engelhardt wrote: Hmm. LZO is the best compression algorithm for the task as measured by the objectives of good compression effectiveness while still having very low CPU usage (the best of those written and GPL'd, there is a slightly better one which is proprietary and uses more CPU, LZRW if I remember right. The gzip code base uses too much CPU, though I think Edward made I don't think that LZO beats LZF in both speed and compression ratio. LZF is also available under GPL (dual-licensed BSD) and was choosen in favor of LZO for the next generation suspend-to-disk code of the Linux kernel. see: http://www.goof.com/pcg/marc/liblzf.html thanks for the info, we will compare them For Suspend2, we ended up converting the LZF support to a cryptoapi plugin. Is there any chance that you could use cryptoapi modules? We could then have a hope of sharing the support. I am throwing in gzip: would it be meaningful to use that instead? The decoder (inflate.c) is already there. 06:04 shanghai:~/liblzf-1.6 l configure* -rwxr-xr-x 1 jengelh users 154894 Mar 3 2005 configure -rwxr-xr-x 1 jengelh users 26810 Mar 3 2005 configure.bz2 -rw-r--r-- 1 jengelh users 30611 Aug 28 20:32 configure.gz-z9 -rw-r--r-- 1 jengelh users 30693 Aug 28 20:32 configure.gz-z6 -rw-r--r-- 1 jengelh users 53077 Aug 28 20:32 configure.lzf We used gzip when we first implemented compression support, and found it to be far too slow. Even with the fastest compression options, we were only getting a few megabytes per second. Perhaps I did something wrong in configuring it, but there's not that many things to get wrong! All that comes to mind is the speed/quality setting -- the number from 1 to 9. Recently, I backed up someone's hard drive using -1, and I believe I was still able to saturate... the _network_. Definitely try again if you haven't changed this, but I can't imagine I'm the first persson to think of it. From what I remember, gzip -1 wasn't faster than the disk. But at least for (very) repetitive data, I was wrong: eve:~ sanity$ time bash -c 'dd if=/dev/zero of=test bs=10m count=10; sync' 10+0 records in 10+0 records out 104857600 bytes transferred in 3.261990 secs (32145287 bytes/sec) real0m3.746s user0m0.005s sys 0m0.627s eve:~ sanity$ time bash -c 'dd if=/dev/zero bs=10m count=10 | gzip -v1 test; sync' 10+0 records in 10+0 records out 104857600 bytes transferred in 2.404093 secs (43616282 bytes/sec) 99.5% real0m2.558s user0m1.554s sys 0m0.680s eve:~ sanity$ This was on OS X, but I think it's still valid -- this is a slightly older Powerbook, with a 5400 RPM drive, 1.6 ghz G4. -1 is still worlds better than nothing. The backup was over 15 gigs, down to about 6 -- loads of repetitive data, I'm sure, but that's where you win with compression anyway. Well, you use cryptoapi anyway, so it should be easy to just let the user pick a plugin, right?
Re: Reiser4 und LZO compression
Nigel Cunningham wrote: Hi. On Mon, 2006-08-28 at 22:15 +0400, Edward Shishkin wrote: Stefan Traby wrote: On Mon, Aug 28, 2006 at 10:06:46AM -0700, Hans Reiser wrote: Hmm. LZO is the best compression algorithm for the task as measured by the objectives of good compression effectiveness while still having very low CPU usage (the best of those written and GPL'd, there is a slightly better one which is proprietary and uses more CPU, LZRW if I remember right. The gzip code base uses too much CPU, though I think Edward made I don't think that LZO beats LZF in both speed and compression ratio. LZF is also available under GPL (dual-licensed BSD) and was choosen in favor of LZO for the next generation suspend-to-disk code of the Linux kernel. see: http://www.goof.com/pcg/marc/liblzf.html thanks for the info, we will compare them For Suspend2, we ended up converting the LZF support to a cryptoapi plugin. Is there any chance that you could use cryptoapi modules? We could then have a hope of sharing the support. No problems with using crypto-api. Reiser4 bypasses it, because currently it supplies the only compression level, which is fairly bad for compressed file systems. Edward.
mutt tells me: Fetching message... Bus error
Hi. I don't know if this can help you to improve reiser4: On a: Linux jolie 2.6.17.8-reiser4-r3-jolie #1 Tue Aug 15 00:14:45 CEST 2006 i686 Intel(R) Pentium(R) 4 CPU 2.53GHz GNU/Linux I was opening a maildir-email-file in mutt via IMAP-SSL running on the same machine. The home and root partition on this machine are reiser4. Mutt did NOT open the Email, but told me: Fetching message... Bus error in the status line. The kernel-log from this time told me: Aug 29 09:55:26 [kernel] [17183751.088000] reiser4[mutt(8457)]: present_lw_sd (fs/reiser4/plugin/item/static_stat.c:277)[]: Aug 29 09:55:26 [kernel] [17183751.088000] WARNING: partially converted file is encountered Aug 29 09:55:26 [imapd-ssl] Unexpected SSL connection shutdown. Aug 29 09:55:26 [imapd-ssl] DISCONNECTED, user=me, ip=[:::127.0.0.1], headers=16823, body=254592, time=3339, starttls=1 Aug 29 09:55:26 [kernel] [17183751.132000] reiser4[mutt(8457)]: extent2tail (fs/reiser4/plugin/file/tail_conversion.c:714)[nikita-2282]: Aug 29 09:55:26 [kernel] [17183751.132000] WARNING: Partial conversion of 2413985: 2 of 3: -22 Aug 29 09:55:26 [kernel] [17183751.132000] reiser4[mutt(8457)]: release_unix_file (fs/reiser4/plugin/file/file.c:2268)[nikita-3233]: Aug 29 09:55:26 [kernel] [17183751.132000] WARNING: Failed (-22) to convert in release_unix_file (2413985) Aug 29 09:55:29 [imapd-ssl] Connection, ip=[:::127.0.0.1] Aug 29 09:55:30 [mutt] unable to dlopen /usr/lib/sasl2/libsql.so: /usr/lib/sasl2/libsql.so: cannot open shared object file: No such file or directory Aug 29 09:55:30 [imapd-ssl] LOGIN, user=me, ip=[:::127.0.0.1], protocol=IMAP Aug 29 09:55:34 [kernel] [17183758.976000] reiser4[mutt(9207)]: extent2tail (fs/reiser4/plugin/file/tail_conversion.c:714)[nikita-2282]: Aug 29 09:55:34 [kernel] [17183758.976000] WARNING: Partial conversion of 2413985: 2 of 3: -22 Aug 29 09:55:34 [kernel] [17183758.976000] reiser4[mutt(9207)]: release_unix_file (fs/reiser4/plugin/file/file.c:2268)[nikita-3233]: Aug 29 09:55:34 [kernel] [17183758.976000] WARNING: Failed (-22) to convert in release_unix_file (2413985) Aug 29 09:55:34 [imapd-ssl] Unexpected SSL connection shutdown. Aug 29 09:55:34 [imapd-ssl] DISCONNECTED, user=me, ip=[:::127.0.0.1], headers=15902, body=3979, time=4, starttls=1 Aug 29 09:58:59 [imapd-ssl] Connection, ip=[:::10.10.10.28] Aug 29 09:58:59 [imapd-ssl] LOGIN, user=me, ip=[:::10.10.10.28], protocol=IMAP Even though with thunderbird via IMAP-SSL from another computer I could open this message! After I booted with the www.sysresccd.org v0.2.19 and did a fsck.reiser4 on this partition. It said I should run fsck.reiser4 --build-fs. Unfortunatly I don't have the output of this call (next time :-). Then I booted into my gentoo linux again ... and now the mail opens in mutt without error message. Greetings, K. Posern -- +-+ K. Posern voicebox phone +49 (0)6421 - 13 5 75 mobile phone +49 (0)6421 - 917 963 +-+ Ich bin eine Fusszeile. Wenn ich gross bin, moechte ich ein Roman werden. pgpkA40FsaCZx.pgp Description: PGP signature
rsync from reiser4 to ext3 caused 100% cpu load and unstoppable task :-)
Hi. I don't know if this can help you to improve reiser4: On a: Linux jolie 2.6.17.8-reiser4-r3-jolie #1 Tue Aug 15 00:14:45 CEST 2006 i686 Intel(R) Pentium(R) 4 CPU 2.53GHz GNU/Linux I did a: rsync -avx --delete-before // /mnt/backup/root/ And as usual I had 3 rsync processes. One of them was in R+ state and at 99%/100% CPU load, the other two were sleeping. I waited the first time for 20h and no change and rsync was still at the same file: etc/mldonkey/actual/donkey.ini.tmp So I decided to STOP this rsync: I tried to CTRL-C the rsync. It did not work. killall rsync did not work. killall -9 rsync, killed one process and left one zombie process (son of the 99%-running process), but the running process was STILL in R+ running ... init 6 told me that it was unable to remount / ro, because of my rsync process :-) So I had to pull the plug to stop it ... The kernel-log from this time told me: Aug 29 09:00:11 [kernel] [17180436.096000] kjournald starting. Commit interval 5 seconds Aug 29 09:00:11 [kernel] [17180436.096000] EXT3-fs warning: maximal mount count reached, running e2fsck is recommended Aug 29 09:00:11 [kernel] [17180436.104000] EXT3 FS on sde6, internal journal Aug 29 09:00:11 [kernel] [17180436.104000] EXT3-fs: mounted filesystem with ordered data mode. Aug 29 09:00:12 [kernel] [17180437.604000] reiser4[rsync(8618)]: present_lw_sd (fs/reiser4/plugin/item/static_stat.c:277)[]: Aug 29 09:00:12 [kernel] [17180437.604000] WARNING: partially converted file is encountered Aug 29 09:00:12 [kernel] [17180437.604000] reiser4[rsync(8618)]: present_lw_sd (fs/reiser4/plugin/item/static_stat.c:277)[]: Aug 29 09:00:12 [kernel] [17180437.604000] WARNING: partially converted file is encountered Aug 29 09:00:12 [kernel] [17180437.608000] reiser4[rsync(8618)]: present_lw_sd (fs/reiser4/plugin/item/static_stat.c:277)[]: Aug 29 09:00:12 [kernel] [17180437.608000] WARNING: partially converted file is encountered Aug 29 09:00:12 [kernel] [17180437.608000] reiser4[rsync(8618)]: present_lw_sd (fs/reiser4/plugin/item/static_stat.c:277)[]: Aug 29 09:00:12 [kernel] [17180437.608000] WARNING: partially converted file is encountered Aug 29 09:00:12 [kernel] [17180437.608000] reiser4[rsync(8618)]: present_lw_sd (fs/reiser4/plugin/item/static_stat.c:277)[]: Aug 29 09:00:12 [kernel] [17180437.608000] WARNING: partially converted file is encountered Aug 29 09:00:12 [kernel] [17180437.608000] reiser4[rsync(8618)]: present_lw_sd (fs/reiser4/plugin/item/static_stat.c:277)[]: Aug 29 09:00:12 [kernel] [17180437.608000] WARNING: partially converted file is encountered Aug 29 09:00:12 [kernel] [17180437.616000] reiser4[rsync(8618)]: present_lw_sd (fs/reiser4/plugin/item/static_stat.c:277)[]: Aug 29 09:00:12 [kernel] [17180437.616000] WARNING: partially converted file is encountered Aug 29 09:00:23 [su] Successful su for root by me Aug 29 09:00:23 [su] + pts/6 me:root Aug 29 09:00:23 [su(pam_unix)] session opened for user root by (uid=1001) Aug 29 09:00:26 [kernel] [17180451.10] reiser4[rsync(8618)]: present_lw_sd (fs/reiser4/plugin/item/static_stat.c:277)[]: Aug 29 09:00:26 [kernel] [17180451.10] WARNING: partially converted file is encountered This behaviour was reproducable after reboot. Then I booted with the www.sysresccd.org v0.2.19 and did a fsck.reiser4 on this partition. It said I should run fsck.reiser4 --build-fs. Unfortunatly I don't have the output of this call (next time :-). Then I booted into my gentoo linux again ... and now the rsync works fine. Greetings, K. Posern -- +-+ K. Posern voicebox phone +49 (0)6421 - 13 5 75 mobile phone +49 (0)6421 - 917 963 +-+ Ich bin eine Fusszeile. Wenn ich gross bin, moechte ich ein Roman werden. -- +-+ K. Posern voicebox phone +49 (0)6421 - 13 5 75 mobile phone +49 (0)6421 - 917 963 +-+ Ich bin eine Fusszeile. Wenn ich gross bin, moechte ich ein Roman werden. pgpugMERh485c.pgp Description: PGP signature
Re: Reiser4 und LZO compression
Hi. On Tue, 2006-08-29 at 03:23 -0500, David Masover wrote: Nigel Cunningham wrote: Hi. On Tue, 2006-08-29 at 06:05 +0200, Jan Engelhardt wrote: Hmm. LZO is the best compression algorithm for the task as measured by the objectives of good compression effectiveness while still having very low CPU usage (the best of those written and GPL'd, there is a slightly better one which is proprietary and uses more CPU, LZRW if I remember right. The gzip code base uses too much CPU, though I think Edward made I don't think that LZO beats LZF in both speed and compression ratio. LZF is also available under GPL (dual-licensed BSD) and was choosen in favor of LZO for the next generation suspend-to-disk code of the Linux kernel. see: http://www.goof.com/pcg/marc/liblzf.html thanks for the info, we will compare them For Suspend2, we ended up converting the LZF support to a cryptoapi plugin. Is there any chance that you could use cryptoapi modules? We could then have a hope of sharing the support. I am throwing in gzip: would it be meaningful to use that instead? The decoder (inflate.c) is already there. 06:04 shanghai:~/liblzf-1.6 l configure* -rwxr-xr-x 1 jengelh users 154894 Mar 3 2005 configure -rwxr-xr-x 1 jengelh users 26810 Mar 3 2005 configure.bz2 -rw-r--r-- 1 jengelh users 30611 Aug 28 20:32 configure.gz-z9 -rw-r--r-- 1 jengelh users 30693 Aug 28 20:32 configure.gz-z6 -rw-r--r-- 1 jengelh users 53077 Aug 28 20:32 configure.lzf We used gzip when we first implemented compression support, and found it to be far too slow. Even with the fastest compression options, we were only getting a few megabytes per second. Perhaps I did something wrong in configuring it, but there's not that many things to get wrong! All that comes to mind is the speed/quality setting -- the number from 1 to 9. Recently, I backed up someone's hard drive using -1, and I believe I was still able to saturate... the _network_. Definitely try again if you haven't changed this, but I can't imagine I'm the first persson to think of it. From what I remember, gzip -1 wasn't faster than the disk. But at least for (very) repetitive data, I was wrong: eve:~ sanity$ time bash -c 'dd if=/dev/zero of=test bs=10m count=10; sync' 10+0 records in 10+0 records out 104857600 bytes transferred in 3.261990 secs (32145287 bytes/sec) real0m3.746s user0m0.005s sys 0m0.627s eve:~ sanity$ time bash -c 'dd if=/dev/zero bs=10m count=10 | gzip -v1 test; sync' 10+0 records in 10+0 records out 104857600 bytes transferred in 2.404093 secs (43616282 bytes/sec) 99.5% real0m2.558s user0m1.554s sys 0m0.680s eve:~ sanity$ This was on OS X, but I think it's still valid -- this is a slightly older Powerbook, with a 5400 RPM drive, 1.6 ghz G4. -1 is still worlds better than nothing. The backup was over 15 gigs, down to about 6 -- loads of repetitive data, I'm sure, but that's where you win with compression anyway. Wow. That's a lot better; I guess I did get something wrong in trying to tune deflate. That was pre-cryptoapi though; looking at cryptoapi/deflate.c, I don't see any way of controlling the compression level. Am I missing anything? Well, you use cryptoapi anyway, so it should be easy to just let the user pick a plugin, right? Right. They can already pick deflate if they want to. Regards, Nigel
Re: Reiser4 und LZO compression
On 8/29/06, Nigel Cunningham [EMAIL PROTECTED] wrote: Hi. On Tue, 2006-08-29 at 03:23 -0500, David Masover wrote: Nigel Cunningham wrote: We used gzip when we first implemented compression support, and found it to be far too slow. Even with the fastest compression options, we were only getting a few megabytes per second. Perhaps I did something wrong in configuring it, but there's not that many things to get wrong! All that comes to mind is the speed/quality setting -- the number from 1 to 9. Recently, I backed up someone's hard drive using -1, and I believe I was still able to saturate... the _network_. Definitely try again if you haven't changed this, but I can't imagine I'm the first persson to think of it. From what I remember, gzip -1 wasn't faster than the disk. But at least for (very) repetitive data, I was wrong: eve:~ sanity$ time bash -c 'dd if=/dev/zero of=test bs=10m count=10; sync' 10+0 records in 10+0 records out 104857600 bytes transferred in 3.261990 secs (32145287 bytes/sec) real0m3.746s user0m0.005s sys 0m0.627s eve:~ sanity$ time bash -c 'dd if=/dev/zero bs=10m count=10 | gzip -v1 test; sync' 10+0 records in 10+0 records out 104857600 bytes transferred in 2.404093 secs (43616282 bytes/sec) 99.5% real0m2.558s user0m1.554s sys 0m0.680s eve:~ sanity$ This was on OS X, but I think it's still valid -- this is a slightly older Powerbook, with a 5400 RPM drive, 1.6 ghz G4. -1 is still worlds better than nothing. The backup was over 15 gigs, down to about 6 -- loads of repetitive data, I'm sure, but that's where you win with compression anyway. Wow. That's a lot better; I guess I did get something wrong in trying to tune deflate. That was pre-cryptoapi though; looking at cryptoapi/deflate.c, I don't see any way of controlling the compression level. Am I missing anything? Compressing /dev/zero isn't a great test. The timings are really data-dependant: [EMAIL PROTECTED]:~$ time bash -c 'sudo dd if=/dev/zero bs=8M count=64 | gzip -v1 /dev/null' 64+0 records in 64+0 records out 536870912 bytes (537 MB) copied, 7.60817 seconds, 70.6 MB/s 99.6% real0m7.652s user0m6.581s sys 0m0.701s [EMAIL PROTECTED]:~$ time bash -c 'sudo dd if=/dev/mem bs=8M count=64 | gzip -v1 /dev/null' 64+0 records in 64+0 records out 536870912 bytes (537 MB) copied, 21.5863 seconds, 24.9 MB/s 70.4% real0m21.626s user0m18.763s sys 0m1.762s This is on an AMD64 laptop. Ray
Re: Reiser4 und LZO compression
Nigel Cunningham wrote: Hi. On Tue, 2006-08-29 at 03:23 -0500, David Masover wrote: Nigel Cunningham wrote: Hi. On Tue, 2006-08-29 at 06:05 +0200, Jan Engelhardt wrote: Hmm. LZO is the best compression algorithm for the task as measured by the objectives of good compression effectiveness while still having very low CPU usage (the best of those written and GPL'd, there is a slightly better one which is proprietary and uses more CPU, LZRW if I remember right. The gzip code base uses too much CPU, though I think Edward made I don't think that LZO beats LZF in both speed and compression ratio. LZF is also available under GPL (dual-licensed BSD) and was choosen in favor of LZO for the next generation suspend-to-disk code of the Linux kernel. see: http://www.goof.com/pcg/marc/liblzf.html thanks for the info, we will compare them For Suspend2, we ended up converting the LZF support to a cryptoapi plugin. Is there any chance that you could use cryptoapi modules? We could then have a hope of sharing the support. I am throwing in gzip: would it be meaningful to use that instead? The decoder (inflate.c) is already there. 06:04 shanghai:~/liblzf-1.6 l configure* -rwxr-xr-x 1 jengelh users 154894 Mar 3 2005 configure -rwxr-xr-x 1 jengelh users 26810 Mar 3 2005 configure.bz2 -rw-r--r-- 1 jengelh users 30611 Aug 28 20:32 configure.gz-z9 -rw-r--r-- 1 jengelh users 30693 Aug 28 20:32 configure.gz-z6 -rw-r--r-- 1 jengelh users 53077 Aug 28 20:32 configure.lzf We used gzip when we first implemented compression support, and found it to be far too slow. Even with the fastest compression options, we were only getting a few megabytes per second. Perhaps I did something wrong in configuring it, but there's not that many things to get wrong! All that comes to mind is the speed/quality setting -- the number from 1 to 9. Recently, I backed up someone's hard drive using -1, and I believe I was still able to saturate... the _network_. Definitely try again if you haven't changed this, but I can't imagine I'm the first persson to think of it. From what I remember, gzip -1 wasn't faster than the disk. But at least for (very) repetitive data, I was wrong: eve:~ sanity$ time bash -c 'dd if=/dev/zero of=test bs=10m count=10; sync' 10+0 records in 10+0 records out 104857600 bytes transferred in 3.261990 secs (32145287 bytes/sec) real0m3.746s user0m0.005s sys 0m0.627s eve:~ sanity$ time bash -c 'dd if=/dev/zero bs=10m count=10 | gzip -v1 test; sync' 10+0 records in 10+0 records out 104857600 bytes transferred in 2.404093 secs (43616282 bytes/sec) 99.5% real0m2.558s user0m1.554s sys 0m0.680s eve:~ sanity$ This was on OS X, but I think it's still valid -- this is a slightly older Powerbook, with a 5400 RPM drive, 1.6 ghz G4. -1 is still worlds better than nothing. The backup was over 15 gigs, down to about 6 -- loads of repetitive data, I'm sure, but that's where you win with compression anyway. Wow. That's a lot better; I guess I did get something wrong in trying to tune deflate. That was pre-cryptoapi though; looking at cryptoapi/deflate.c, I don't see any way of controlling the compression level. Am I missing anything? zlib is tunable, not cryptoapi's deflate. look at zlib_deflateInit2() Well, you use cryptoapi anyway, so it should be easy to just let the user pick a plugin, right? Right. They can already pick deflate if they want to. Regards, Nigel
Re: Reiser4 und LZO compression
Would it be, by any chance, possible to tweak the thing so that reiserfs plugins become kernel modules, so that the reiserfs core can be put in the kernel without the plugins slowing down its acceptance ? (and updating plugins without rebooting would be a nice extra) The patch below is so-called reiser4 LZO compression plugin as extracted from 2.6.18-rc4-mm3. I think it is an unauditable piece of shit and thus should not enter mainline. Like lib/inflate.c (and this new code should arguably be in lib/). The problem is that if we clean this up, we've diverged very much from the upstream implementation. So taking in fixes and features from upstream becomes harder and more error-prone. I'd suspect that the maturity of these utilities is such that we could afford to turn them into kernel code in the expectation that any future changes will be small. But it's not a completely simple call. (iirc the inflate code had a buffer overrun a while back, which was found and fixed in the upstream version).
Re: Reiser4 und LZO compression
On Tue, Aug 29, 2006 at 03:45:59PM +0200, PFC wrote: Anyone has a bench for lzf ? It's easy, try something like: wget http://www.goof.com/pcg/marc/data/liblzf-1.6.tar.gz tar zxvpf liblzf-1.6.tar.gz cd liblzf-1.6 configure make Now you have a small lzf binary that you can use for testing: cat bigfile|./lzf bigfile.lzf use ./lzf -d for decompression tests -- ciao - Stefan
Re: Reiser4 und LZO compression
On 8/29/06, PFC [EMAIL PROTECTED] wrote: Anyone has a bench for lzf ? This is on a opteron 1.8GHz box. Everything tested hot cache. Testing on a fairly repetative but real test case (an SQL dump of one of the Wikipedia tables): -rw-rw-r-- 1 gmaxwell gmaxwell 426162134 Jul 20 06:54 ../page.sql $time lzop -c ../page.sql page.sql.lzo real0m8.618s user0m7.800s sys 0m0.808s $time lzop -9c ../page.sql page.sql.lzo-9 real4m45.299s user4m44.474s sys 0m0.712s $time gzip -1 -c ../page.sql page.sql.gz real0m19.292s user0m18.545s sys 0m0.748s $time lzop -d -c ./page.sql.lzo /dev/null real0m3.061s user0m2.836s sys 0m0.224s $time gzip -dc page.sql.gz /dev/null real0m7.199s user0m7.020s sys 0m0.176s $time ./lzf -d page.sql.lzf /dev/null real0m2.398s user0m2.224s sys 0m0.172s -rw-rw-r-- 1 gmaxwell gmaxwell 193853815 Aug 29 10:59 page.sql.gz -rw-rw-r-- 1 gmaxwell gmaxwell 243497298 Aug 29 10:47 page.sql.lzf -rw-rw-r-- 1 gmaxwell gmaxwell 259986955 Jul 20 06:54 page.sql.lzo -rw-rw-r-- 1 gmaxwell gmaxwell 204930904 Jul 20 06:54 page.sql.lzo-9 (decompression of the differing lzo levels is the same speed) None of them really decompress fast enough to keep up with the disks in this system, lzf or lzo wouldn't be a big loss. (Bonnie scores: floodlamp,64G,,,246163,52,145536,35,,,365198,42,781.2,2,16,4540,69,+,+++,2454,31,4807,76,+,+++,2027,36)
Re: Reiser4 und LZO compression
I have made a little openoffice spreadsheet with the results. You can have fun entering stuff and seeing the results. http://peufeu.free.fr/compression.ods Basically, a laptop having the same processor as my PC and a crummy 15 MB/s drive (like most laptop drives) will get a 2.5x speedup using lzf, while using 40% CPU for compression and 15% CPU for decompression. I'd say it's a clear, hge win. A desktop computer with a modern IDE drive doing 50 MB/s will still get nice speedups (1.8x on write, 2.5x on read) but of course, more CPU will be used because of the higher throughput. In this case it is CPU limited on compression and disk limited on decompression. However soon everyone will have dual core monsters so... A big ass RAID will not get much benefit unless : - the buffer cache stores compressed pages, so compression virtually doubles the RAM cache - or the CPU is really fast - or you put one of these neat FPGA modules in a free Opteron socket and upload a soft-hardware LZF in it with a few gigabytes/s throughput ...
Re: Reiser4 und LZO compression
PFC wrote: Would it be, by any chance, possible to tweak the thing so that reiserfs plugins become kernel modules, so that the reiserfs core can be put in the kernel without the plugins slowing down its acceptance ? I don't see what this has to do with cryptoapi plugins -- those are not related to Reiser plugins. As for the plugins slowing down acceptance, it's actually the concept of plugins and the plugin API -- in other words, it's the fact that Reiser4 supports plugins -- that is slowing it down, if anything about plugins is still an issue at all. Making them modules would make it worse. Last I saw, Linus doesn't particularly like the idea of plugins because of a few misconceptions, like the possibility of proprietary (possibly GPL-violating) plugins distributed as modules -- basically, something like what nVidia and ATI do with their video drivers. As it is, a good argument in favor of plugins is that this kind of thing isn't possible -- we often put plugins in quotes because really, it's just a nice abstraction layer. They aren't any more plugins than iptables modules or cryptoapi plugins are. If anything, they're less, because they must be compiled into Reiser4, which means either one huge monolithic Reiser4 module (including all plugins), or everything compiled into the kernel image. (and updating plugins without rebooting would be a nice extra) It probably wouldn't be as nice as you think. Remember, if you're using a certain plugin in your root FS, it's part of the FS, so I don't think you'd be able to remove that plugin any more than you're able to remove reiser4.ko if that's your root FS. You'd have to unmount every FS that uses that plugin. At this point, you don't really gain much -- if you unmount every last Reiser4 filesystem, you can then remove reiser4.ko, recompile it, and load a new one with different plugins enabled. Also, these things would typically be part of a kernel update anyway, meaning a reboot anyway. But suppose you could remove a plugin, what then? What would that mean? Suppose half your files are compressed and you remove cryptocompress -- are those files uncompressed when the plugin goes away? Probably not. The only smart way to handle this that I can think of is to make those files unavailable, which is probably not what you want -- how do you update cryptocompress when the new reiser4_cryptocompress.ko is itself compressed? That may be an acceptable solution for some plugins, but you'd have to be extremely careful which ones you remove. The only safe way I can imagine doing this may not be possible, and if it is, it's extremely hackish -- load the plugin under another module name, so r4_cryptocompress would be r4_cryptocompress_init -- have the module, once loaded, do an atomic switch from the old one to the new one, effectively in-place. But that kind of solution is something I've never seen attempted, and only really heard of in strange environments like Erlang. It would probably require much more engineering than the Reiser team can handle right now, especially with their hands full with inclusion. The patch below is so-called reiser4 LZO compression plugin as extracted from 2.6.18-rc4-mm3. I think it is an unauditable piece of shit and thus should not enter mainline. Like lib/inflate.c (and this new code should arguably be in lib/). The problem is that if we clean this up, we've diverged very much from the upstream implementation. So taking in fixes and features from upstream becomes harder and more error-prone. I'd suspect that the maturity of these utilities is such that we could afford to turn them into kernel code in the expectation that any future changes will be small. But it's not a completely simple call. (iirc the inflate code had a buffer overrun a while back, which was found and fixed in the upstream version).
Re: Reiser4 und LZO compression
PFC wrote: I made a little benchmark on my own PC (Athlon64 3200+ in 64 bit gentoo) http://peufeu.free.fr/compression.html So, gzip could be used on PCs having very fast processors and very slow harddrives, like Core Duo laptops. However, lzo compresses nearly as much and is still a lot faster. I don't see a reason for gzip in a FS application. Anyone has a bench for lzf ? Yes, Edward did equivalent tests, and thus we selected LZO.
Re: Reiser4 und LZO compression
PFC, thanks for giving us some real data. May I post it to the lkml thread? In essence, LZO wins the benchmarks, and the code is hard to read. I guess I have to go with LZO, and encourage people to take a stab at dethroning it. Hans PFC wrote: I have made a little openoffice spreadsheet with the results. You can have fun entering stuff and seeing the results. http://peufeu.free.fr/compression.ods Basically, a laptop having the same processor as my PC and a crummy 15 MB/s drive (like most laptop drives) will get a 2.5x speedup using lzf, while using 40% CPU for compression and 15% CPU for decompression. I'd say it's a clear, hge win. A desktop computer with a modern IDE drive doing 50 MB/s will still get nice speedups (1.8x on write, 2.5x on read) but of course, more CPU will be used because of the higher throughput. In this case it is CPU limited on compression and disk limited on decompression. However soon everyone will have dual core monsters so... A big ass RAID will not get much benefit unless : - the buffer cache stores compressed pages, so compression virtually doubles the RAM cache - or the CPU is really fast - or you put one of these neat FPGA modules in a free Opteron socket and upload a soft-hardware LZF in it with a few gigabytes/s throughput Or you look the sysadmin in the eyes, and say, your file servers have more out of disk space problems than load problems, yes? ...
Re: mutt tells me: Fetching message... Bus error
Thanks. Did you run an older version of reiser4 before this? If yes, then this may have been fixed but only showed up for you now. Zam? Hans Posern wrote: Hi. I don't know if this can help you to improve reiser4: On a: Linux jolie 2.6.17.8-reiser4-r3-jolie #1 Tue Aug 15 00:14:45 CEST 2006 i686 Intel(R) Pentium(R) 4 CPU 2.53GHz GNU/Linux I was opening a maildir-email-file in mutt via IMAP-SSL running on the same machine. The home and root partition on this machine are reiser4. Mutt did NOT open the Email, but told me: Fetching message... Bus error in the status line. The kernel-log from this time told me: Aug 29 09:55:26 [kernel] [17183751.088000] reiser4[mutt(8457)]: present_lw_sd (fs/reiser4/plugin/item/static_stat.c:277)[]: Aug 29 09:55:26 [kernel] [17183751.088000] WARNING: partially converted file is encountered Aug 29 09:55:26 [imapd-ssl] Unexpected SSL connection shutdown. Aug 29 09:55:26 [imapd-ssl] DISCONNECTED, user=me, ip=[:::127.0.0.1], headers=16823, body=254592, time=3339, starttls=1 Aug 29 09:55:26 [kernel] [17183751.132000] reiser4[mutt(8457)]: extent2tail (fs/reiser4/plugin/file/tail_conversion.c:714)[nikita-2282]: Aug 29 09:55:26 [kernel] [17183751.132000] WARNING: Partial conversion of 2413985: 2 of 3: -22 Aug 29 09:55:26 [kernel] [17183751.132000] reiser4[mutt(8457)]: release_unix_file (fs/reiser4/plugin/file/file.c:2268)[nikita-3233]: Aug 29 09:55:26 [kernel] [17183751.132000] WARNING: Failed (-22) to convert in release_unix_file (2413985) Aug 29 09:55:29 [imapd-ssl] Connection, ip=[:::127.0.0.1] Aug 29 09:55:30 [mutt] unable to dlopen /usr/lib/sasl2/libsql.so: /usr/lib/sasl2/libsql.so: cannot open shared object file: No such file or directory Aug 29 09:55:30 [imapd-ssl] LOGIN, user=me, ip=[:::127.0.0.1], protocol=IMAP Aug 29 09:55:34 [kernel] [17183758.976000] reiser4[mutt(9207)]: extent2tail (fs/reiser4/plugin/file/tail_conversion.c:714)[nikita-2282]: Aug 29 09:55:34 [kernel] [17183758.976000] WARNING: Partial conversion of 2413985: 2 of 3: -22 Aug 29 09:55:34 [kernel] [17183758.976000] reiser4[mutt(9207)]: release_unix_file (fs/reiser4/plugin/file/file.c:2268)[nikita-3233]: Aug 29 09:55:34 [kernel] [17183758.976000] WARNING: Failed (-22) to convert in release_unix_file (2413985) Aug 29 09:55:34 [imapd-ssl] Unexpected SSL connection shutdown. Aug 29 09:55:34 [imapd-ssl] DISCONNECTED, user=me, ip=[:::127.0.0.1], headers=15902, body=3979, time=4, starttls=1 Aug 29 09:58:59 [imapd-ssl] Connection, ip=[:::10.10.10.28] Aug 29 09:58:59 [imapd-ssl] LOGIN, user=me, ip=[:::10.10.10.28], protocol=IMAP Even though with thunderbird via IMAP-SSL from another computer I could open this message! After I booted with the www.sysresccd.org v0.2.19 and did a fsck.reiser4 on this partition. It said I should run fsck.reiser4 --build-fs. Unfortunatly I don't have the output of this call (next time :-). Then I booted into my gentoo linux again ... and now the mail opens in mutt without error message. Greetings, K. Posern
Re: Reiser4 und LZO compression
On 8/29/06, David Masover [EMAIL PROTECTED] wrote: [snip] Conversely, compression does NOT make sense if: - You spend a lot of time with the CPU busy and the disk idle. - You have more than enough disk space. - Disk space is cheaper than buying enough CPU to handle compression. - You've tried compression, and the CPU requirements slowed you more than you saved in disk access. [snip] It's also not always this simple ... if you have a single threaded workload that doesn't overlap CPU and disk well, (de)compression may be free even if you're still CPU bound a lot as the compression is using cpu cycles which would have been otherwise idle..
Re: [2006-08#14.139] Re: mutt tells me: Fetching message... Bus error
Hi. I am using reiser4 since not so long. In my mind it is only a few month. I verified and I think my first kernel with reiser4 was: linux-2.6.16.11-reiser4-r2 and now it is: linux-2.6.17.8-reiser4-r3 I am using a gentoo based system. And the package system portage logs nice things: jolie ~ # genlop -e reiser4progs * sys-fs/reiser4progs Fri Nov 12 15:44:41 2004 sys-fs/reiser4progs-1.0.1 Sun Jan 22 05:59:28 2006 sys-fs/reiser4progs-1.0.5 So it means that from the beginning I used reiser4progs v1.0.5. --- For the kernels I am using a selfbuilt ebuild to autoamtically patch the vanilla kernel with the newest according patch from your ftp server. And for this ebuild, my gentoo logs say: * sys-kernel/reiser4-sources Wed Apr 12 20:55:39 2006 sys-kernel/reiser4-sources-2.6.16.4-r2 Mon May 1 19:12:11 2006 sys-kernel/reiser4-sources-2.6.16.11-r2 Mon Aug 14 12:15:16 2006 sys-kernel/reiser4-sources-2.6.17.8-r3 So maybe I also used already reiser4-sources-2.6.16.4-r2... ...but I think I only downloaded this package and I did NOT configure, build and actually RUN this version. ...but maybe I also deleted just this kernel-config :-) --- Thats all I can tell. Hope it helps. Greetings, K. Posern = On Tue, Aug 29, 2006 at 11:12:01AM -0700, Hans Reiser wrote: Thanks. Did you run an older version of reiser4 before this? If yes, then this may have been fixed but only showed up for you now. Zam? Hans Posern wrote: Hi. I don't know if this can help you to improve reiser4: On a: Linux jolie 2.6.17.8-reiser4-r3-jolie #1 Tue Aug 15 00:14:45 CEST 2006 i686 Intel(R) Pentium(R) 4 CPU 2.53GHz GNU/Linux I was opening a maildir-email-file in mutt via IMAP-SSL running on the same machine. The home and root partition on this machine are reiser4. Mutt did NOT open the Email, but told me: Fetching message... Bus error in the status line. The kernel-log from this time told me: Aug 29 09:55:26 [kernel] [17183751.088000] reiser4[mutt(8457)]: present_lw_sd (fs/reiser4/plugin/item/static_stat.c:277)[]: Aug 29 09:55:26 [kernel] [17183751.088000] WARNING: partially converted file is encountered Aug 29 09:55:26 [imapd-ssl] Unexpected SSL connection shutdown. Aug 29 09:55:26 [imapd-ssl] DISCONNECTED, user=me, ip=[:::127.0.0.1], headers=16823, body=254592, time=3339, starttls=1 Aug 29 09:55:26 [kernel] [17183751.132000] reiser4[mutt(8457)]: extent2tail (fs/reiser4/plugin/file/tail_conversion.c:714)[nikita-2282]: Aug 29 09:55:26 [kernel] [17183751.132000] WARNING: Partial conversion of 2413985: 2 of 3: -22 Aug 29 09:55:26 [kernel] [17183751.132000] reiser4[mutt(8457)]: release_unix_file (fs/reiser4/plugin/file/file.c:2268)[nikita-3233]: Aug 29 09:55:26 [kernel] [17183751.132000] WARNING: Failed (-22) to convert in release_unix_file (2413985) Aug 29 09:55:29 [imapd-ssl] Connection, ip=[:::127.0.0.1] Aug 29 09:55:30 [mutt] unable to dlopen /usr/lib/sasl2/libsql.so: /usr/lib/sasl2/libsql.so: cannot open shared object file: No such file or directory Aug 29 09:55:30 [imapd-ssl] LOGIN, user=me, ip=[:::127.0.0.1], protocol=IMAP Aug 29 09:55:34 [kernel] [17183758.976000] reiser4[mutt(9207)]: extent2tail (fs/reiser4/plugin/file/tail_conversion.c:714)[nikita-2282]: Aug 29 09:55:34 [kernel] [17183758.976000] WARNING: Partial conversion of 2413985: 2 of 3: -22 Aug 29 09:55:34 [kernel] [17183758.976000] reiser4[mutt(9207)]: release_unix_file (fs/reiser4/plugin/file/file.c:2268)[nikita-3233]: Aug 29 09:55:34 [kernel] [17183758.976000] WARNING: Failed (-22) to convert in release_unix_file (2413985) Aug 29 09:55:34 [imapd-ssl] Unexpected SSL connection shutdown. Aug 29 09:55:34 [imapd-ssl] DISCONNECTED, user=me, ip=[:::127.0.0.1], headers=15902, body=3979, time=4, starttls=1 Aug 29 09:58:59 [imapd-ssl] Connection, ip=[:::10.10.10.28] Aug 29 09:58:59 [imapd-ssl] LOGIN, user=me, ip=[:::10.10.10.28], protocol=IMAP Even though with thunderbird via IMAP-SSL from another computer I could open this message! After I booted with the www.sysresccd.org v0.2.19 and did a fsck.reiser4 on this partition. It said I should run fsck.reiser4 --build-fs. Unfortunatly I don't have the output of this call (next time :-). Then I booted into my gentoo linux again ... and now the mail opens in mutt without error message. Greetings, K. Posern -- ++ MPsoft IT Solutions Knuth Posern Weidenhaeuser Str. 26 35037 Marburg Germany Tel. +49 (0) 6421 - 17 68 133 Fax. +49 (0) 6421 - 17 68 134 Mobil +49 (0) 6421 - 917 963 ++ pgpyotK2sgWuy.pgp Description: PGP signature
Re: Reiser4 und LZO compression
Gregory Maxwell wrote: On 8/29/06, David Masover [EMAIL PROTECTED] wrote: [snip] Conversely, compression does NOT make sense if: - You spend a lot of time with the CPU busy and the disk idle. - You have more than enough disk space. - Disk space is cheaper than buying enough CPU to handle compression. - You've tried compression, and the CPU requirements slowed you more than you saved in disk access. [snip] It's also not always this simple ... if you have a single threaded workload that doesn't overlap CPU and disk well, (de)compression may be free even if you're still CPU bound a lot as the compression is using cpu cycles which would have been otherwise idle.. Isn't that implied, though -- if the CPU is not busy (run top under a 2.6 kernel and you'll see an IO-Wait number), then the first condition isn't satisfied -- CPU is not busy, disk is not idle. But speaking of single threadedness, more and more desktops are shipping with ridiculously more power than people need. Even a gamer really won't benefit that much from having a dual-core system, because multithreading is hard, and games haven't been doing it properly. John Carmack is pretty much the only superstar programmer in video games, and after his first fairly massive attempt to make Quake 3 have two threads (since he'd just gotten a dual-core machine to play with) actually resulted in the game running some 30-40% slower than it did with a single thread. So, for the desktop, compression makes perfect sense. We don't have massive amounts of RAID. If we have newer machines, there's a good chance we'll have one CPU sitting mostly idle while playing games. Short of gaming, there are few desktop applications that will fully utilize even one reasonably fast CPU. The reason gamers buy dual-core systems is they're getting cheap enough to be worth it, and that one core sitting idle is a perfect place to do OS/system work not related to the game -- antivirus, automatic update checks, the inevitable background processes leeching a couple few % off your available CPU. So for the typical new desktop with about 2 ghz of 64-bit processor sitting idle, compression is essentially free.
Re: Reiser4 und LZO compression
David Masover wrote: John Carmack is pretty much the only superstar programmer in video games, and after his first fairly massive attempt to make Quake 3 have two threads (since he'd just gotten a dual-core machine to play with) actually resulted in the game running some 30-40% slower than it did with a single thread. Do the two processors have separate caches, and thus being overly fined grained makes you memory transfer bound or? Two processors tends to create a snappier user experience, in that big CPU processes get throttled nicely. Hans
Re: Reiser4 und LZO compression
Hans Reiser wrote: David Masover wrote: John Carmack is pretty much the only superstar programmer in video games, and after his first fairly massive attempt to make Quake 3 have two threads (since he'd just gotten a dual-core machine to play with) actually resulted in the game running some 30-40% slower than it did with a single thread. Do the two processors have separate caches, and thus being overly fined grained makes you memory transfer bound or? It wasn't anything that intelligent. Let me see if I can find it... Taken from http://techreport.com/etc/2005q3/carmack-quakecon/index.x?pg=1 Graphics accelerators are a great example of parallelism working well, he noted, but game code is not similarly parallelizable. Carmack cited his Quake III Arena engine, whose renderer was multithreaded and achieved up to 40% performance increases on multiprocessor systems, as a good example of where games would have to go. (Q3A's SMP mode was notoriously crash-prone and fragile, working only with certain graphics driver revisions and the like.) Initial returns on multithreading, he projected, will be disappointing. Basically, it's hard enough to split what we currently do onto even 2 CPUs, and it definitely seems like we're about to hit a wall in CPU frequency just as multicore becomes a practical reality, so future CPUs may be measured in how many cores they have, not how fast each core is. There's also a question of what to use the extra power for. From the same presentation: Part of the problem with multithreading, argued Carmack, is knowing how to use the power of additional CPU cores to enhance the game experience. A.I., can be effective when very simple, as some of the first Doom logic was. It was less than a page of code, but players ascribed complex behaviors and motivations to the bad guys. However, more complex A.I. seems hard to improve to the point where it really changes the game. More physics detail, meanwhile, threatens to make games too fragile as interactions in the game world become more complex. So, I humbly predict that Physics cards (so-called PPUs) will fail, and be replaced by ever-increasing numbers of cores, which will, for awhile, be one step ahead of what we can think of to fill them with. Thus, anything useful (like compression) that can be split off into a separate thread is going to be useful for games, and won't hurt performance on future mega-multicore monstrosities. The downside is, most game developers are working on Windows, for which FS compression has always sucked. Thus, they most often implement their own compression, often something horrible, like storing the whole game in CAB or ZIP files, and loading the entire level into RAM before play starts, making load times less relevant for gameplay. Reiser4's cryptocompress would be a marked improvement over that, but it would also not be used in many games.
[patch] Re: assertion failed: can_hit_entd(ctx, s)
Hello Alexander,
In addition to your patch, I've also applied the patch below. With
these two patches the fs is much more stable for me.
However, something is holding a d_ref across the calls to
reiser4_writepage. It's not clear to me that this is allowed so my
patch may not be a full fix.
Andrew Wade
signed-off-by: [EMAIL PROTECTED]
diff -rupN a/fs/reiser4/plugin/item/extent_file_ops.c
b/fs/reiser4/plugin/item/extent_file_ops.c
--- a/fs/reiser4/plugin/item/extent_file_ops.c 2006-08-28 11:30:33.0
-0400
+++ b/fs/reiser4/plugin/item/extent_file_ops.c 2006-08-29 13:06:20.0
-0400
@@ -1320,20 +1320,22 @@ static int extent_readpage_filler(void *
TWIG_LEVEL, CBK_UNIQUE, NULL);
if (result != CBK_COORD_FOUND) {
reiser4_unset_hint(hint);
- return result;
+ goto out;
}
ext_coord-valid = 0;
}
if (zload(ext_coord-coord.node)) {
reiser4_unset_hint(hint);
- return RETERR(-EIO);
+ result = RETERR(-EIO);
+ goto out;
}
if (!item_is_extent(ext_coord-coord)) {
/* tail conversion is running in parallel */
zrelse(ext_coord-coord.node);
reiser4_unset_hint(hint);
- return RETERR(-EIO);
+ result = RETERR(-EIO);
+ goto out;
}
if (ext_coord-valid == 0)
@@ -1358,6 +1360,10 @@ static int extent_readpage_filler(void *
} else
reiser4_unset_hint(hint);
zrelse(ext_coord-coord.node);
+
+out:
+ /* Calls to this function may be intermingled with VM writeback. */
+ reiser4_txn_restart_current();
return result;
}
Re: assertion failed: JF_ISSET(jprivate(page), JNODE_DIRTY)
I now have a stack trace for this assertion: reiser4 panicked cowardly: reiser4[tar(5412)]: reiser4_set_page_dirty_internal (fs/reiser4/page_cache.c:475)[]: assertion failed: JF_ISSET(jprivate(page), JNODE_DIRTY) [c0103870] dump_trace+0x64/0x1ad [c01039cb] show_trace_log_lvl+0x12/0x25 [c0103cc1] show_trace+0xd/0x10 [c0103cdb] dump_stack+0x17/0x19 [c01a0caf] reiser4_do_panic+0x4e/0x84 [c01e66e9] reiser4_set_page_dirty_internal+0xe5/0xee [c01cb2a4] znode_make_dirty+0x271/0x452 [c022108f] cut_node40+0x191/0x1a6 [c0221410] shift_node40+0x36c/0x91d [c01b11ba] carry_shift_data+0xaa/0x139 [c01b2e1b] carry_insert_flow+0x1de/0x837 [c01b008f] reiser4_carry+0x185/0x49a [c01b8d77] reiser4_insert_flow+0x16b/0x17e [c022deec] reiser4_write_tail+0x5cd/0x685 [c020445f] batch_write_unix_file+0x26e/0x467 [c0133347] generic_file_buffered_write+0xd2/0x1fb [c0135035] __generic_file_aio_write_nolock+0x3a8/0x3e5 [c01350ca] generic_file_aio_write+0x58/0xab [c014eda6] do_sync_write+0xb4/0xf2 [c014f38e] vfs_write+0x8a/0x136 [c014faca] sys_write+0x3b/0x60 [c01028cd] sysenter_past_esp+0x56/0x8d DWARF2 unwinder stuck at sysenter_past_esp+0x56/0x8d Leftover inexact backtrace: [c01039cb] show_trace_log_lvl+0x12/0x25 [c0103cc1] show_trace+0xd/0x10 [c0103cdb] dump_stack+0x17/0x19 [c01a0caf] reiser4_do_panic+0x4e/0x84 [c01e66e9] reiser4_set_page_dirty_internal+0xe5/0xee [c01cb2a4] znode_make_dirty+0x271/0x452 [c022108f] cut_node40+0x191/0x1a6 [c0221410] shift_node40+0x36c/0x91d [c01b11ba] carry_shift_data+0xaa/0x139 [c01b2e1b] carry_insert_flow+0x1de/0x837 [c01b008f] reiser4_carry+0x185/0x49a [c01b8d77] reiser4_insert_flow+0x16b/0x17e [c022deec] reiser4_write_tail+0x5cd/0x685 [c020445f] batch_write_unix_file+0x26e/0x467 [c0133347] generic_file_buffered_write+0xd2/0x1fb [c0135035] __generic_file_aio_write_nolock+0x3a8/0x3e5 [c01350ca] generic_file_aio_write+0x58/0xab [c014eda6] do_sync_write+0xb4/0xf2 [c014f38e] vfs_write+0x8a/0x136 [c014faca] sys_write+0x3b/0x60 [c01028cd] sysenter_past_esp+0x56/0x8d === : jnode: 0, tree: 0 (r:0,w:0), dk: 0 (r:0,w:0) jload: 0, txnh: 0, atom: 0, stack: 0, txnmgr: 0, ktxnmgrd: 0, fq: 0 inode: 0, cbk_cache: 0 (r:0,w0), eflush: 0, zlock: 0, spin: 0, long: 3 inode_sem: (r:0,w:1) d: 3, x: 6, t: 0 Kernel panic - not syncing: reiser4[tar(5412)]: reiser4_set_page_dirty_internal (fs/reiser4/page_cache.c:475)[]: assertion failed: JF_ISSET(jprivate(page), JNODE_DIRTY) Andrew Wade
Re: Reiser4 und LZO compression
Hi. On Tue, 2006-08-29 at 15:38 +0400, Edward Shishkin wrote: Nigel Cunningham wrote: Hi. On Tue, 2006-08-29 at 03:23 -0500, David Masover wrote: Nigel Cunningham wrote: Hi. On Tue, 2006-08-29 at 06:05 +0200, Jan Engelhardt wrote: Hmm. LZO is the best compression algorithm for the task as measured by the objectives of good compression effectiveness while still having very low CPU usage (the best of those written and GPL'd, there is a slightly better one which is proprietary and uses more CPU, LZRW if I remember right. The gzip code base uses too much CPU, though I think Edward made I don't think that LZO beats LZF in both speed and compression ratio. LZF is also available under GPL (dual-licensed BSD) and was choosen in favor of LZO for the next generation suspend-to-disk code of the Linux kernel. see: http://www.goof.com/pcg/marc/liblzf.html thanks for the info, we will compare them For Suspend2, we ended up converting the LZF support to a cryptoapi plugin. Is there any chance that you could use cryptoapi modules? We could then have a hope of sharing the support. I am throwing in gzip: would it be meaningful to use that instead? The decoder (inflate.c) is already there. 06:04 shanghai:~/liblzf-1.6 l configure* -rwxr-xr-x 1 jengelh users 154894 Mar 3 2005 configure -rwxr-xr-x 1 jengelh users 26810 Mar 3 2005 configure.bz2 -rw-r--r-- 1 jengelh users 30611 Aug 28 20:32 configure.gz-z9 -rw-r--r-- 1 jengelh users 30693 Aug 28 20:32 configure.gz-z6 -rw-r--r-- 1 jengelh users 53077 Aug 28 20:32 configure.lzf We used gzip when we first implemented compression support, and found it to be far too slow. Even with the fastest compression options, we were only getting a few megabytes per second. Perhaps I did something wrong in configuring it, but there's not that many things to get wrong! All that comes to mind is the speed/quality setting -- the number from 1 to 9. Recently, I backed up someone's hard drive using -1, and I believe I was still able to saturate... the _network_. Definitely try again if you haven't changed this, but I can't imagine I'm the first persson to think of it. From what I remember, gzip -1 wasn't faster than the disk. But at least for (very) repetitive data, I was wrong: eve:~ sanity$ time bash -c 'dd if=/dev/zero of=test bs=10m count=10; sync' 10+0 records in 10+0 records out 104857600 bytes transferred in 3.261990 secs (32145287 bytes/sec) real0m3.746s user0m0.005s sys 0m0.627s eve:~ sanity$ time bash -c 'dd if=/dev/zero bs=10m count=10 | gzip -v1 test; sync' 10+0 records in 10+0 records out 104857600 bytes transferred in 2.404093 secs (43616282 bytes/sec) 99.5% real0m2.558s user0m1.554s sys 0m0.680s eve:~ sanity$ This was on OS X, but I think it's still valid -- this is a slightly older Powerbook, with a 5400 RPM drive, 1.6 ghz G4. -1 is still worlds better than nothing. The backup was over 15 gigs, down to about 6 -- loads of repetitive data, I'm sure, but that's where you win with compression anyway. Wow. That's a lot better; I guess I did get something wrong in trying to tune deflate. That was pre-cryptoapi though; looking at cryptoapi/deflate.c, I don't see any way of controlling the compression level. Am I missing anything? zlib is tunable, not cryptoapi's deflate. look at zlib_deflateInit2() Ok; thanks. I wasn't mistaken then :) Regards, Nigel
Re: Reiser4 und LZO compression
On 29-Aug-06, at 4:03 PM, David Masover wrote: Hans Reiser wrote: David Masover wrote: John Carmack is pretty much the only superstar programmer in video games, and after his first fairly massive attempt to make Quake 3 have two threads (since he'd just gotten a dual-core machine to play with) actually resulted in the game running some 30-40% slower than it did with a single thread. Do the two processors have separate caches, and thus being overly fined grained makes you memory transfer bound or? It wasn't anything that intelligent. Let me see if I can find it... Taken from http://techreport.com/etc/2005q3/carmack-quakecon/index.x?pg=1 Graphics accelerators are a great example of parallelism working well, he noted, but game code is not similarly parallelizable. ... The downside is, most game developers are working on Windows, for which FS compression has always sucked. Thus, they most often implement their own compression, often something horrible, like storing the whole game in CAB or ZIP files, and loading the entire level into RAM before play starts, making load times less relevant for gameplay. Reiser4's cryptocompress would be a marked improvement over that, but it would also not be used in many games. Gamer systems, whether from coder's or player's p.o.v., would appear fairly irrelevant to reiserfs and this list. I'd trust Carmack's eye candy credentials but doubt he has much to say about filesystems or server threading...
Re: Reiser4 und LZO compression
Toby Thain wrote: Gamer systems, whether from coder's or player's p.o.v., would appear fairly irrelevant to reiserfs and this list. I'd trust Carmack's eye candy credentials but doubt he has much to say about filesystems or server threading... Maybe, but Reiser4 is supposed to be a general purpose filesystem, so talking about its advantages/disadvantages wrt. gaming makes sense, especially considering gamers are the most likely to tune their desktop for perfomance. That was a bit much, though. I apologize.
