Re: Reiser4 patches
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 reiser4-for-2.6.20.patch works fine now on my setup. No more panics in do_readpage_extent. System is up and running under heavy disk io for 3 days now. Thank you very much, for the great work. regards DevH -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFGRxU4CbmO6P7mspMRCqbQAJ0S1JXYJ25WUMJTh4OYDEZ/cpdMhQCgmADW 7lbiyiCSCNMPenk8Z4Rbvho= =XReE -END PGP SIGNATURE-
Re: reiser4 causes system freeze for 1-2 seconds
Hello Edward, first thanks for your answer. My system has 1536M. After I has started my system with 512M I copied three different files and the system freezed only once for less than 0,5 seconds and ent:hda8! shows up less frequently in top. But after rebooting with 1536M I could not reproduce the longer freezes. So it has to be also related with uptime and programs I'm using. Uptime was 2 days and the most important programs to mention are firefox and azureus. Azureus was downloading to the reiser4 partition but only with 10-15kb/s. I have rebooted with 512M and will try it some days. Thanks Benni On Wed, 04 Apr 2007 03:08:36 +0400 Edward Shishkin [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] wrote: Hello. Hello, I have a 120Gb partition with reiser4. If I start a command like cp 500mb file dd within it the system frequently freezes for 1-2 seconds and ent:hda8! uses nearly 100% cpu. This causes the sound output from xmms to be interrupted and my mouse freezes also. If i renice X and xmms to -10 it is okay. Is this the desired behaviour ? No. Can I do sth. else than renicing all time critical programs ? My system is a debian amd64 with linux-2.6.16-mm2. How much RAM is available in your system? If this is much more then 512M, then boot with mem=512M, and let us know if this freeze still takes place (to make sure this is because of unwisely assigned atom_max_size). Thanks, Edward. Best Regards Benni
Re: reiser4 causes system freeze for 1-2 seconds
After some hours the problem has occured again. with azureus running, almost three second freezed top - 19:28:24 up 8:25, 9 users, load average: 6.89, 2.72, 1.26 Tasks: 110 total, 11 running, 98 sleeping, 0 stopped, 1 zombie Cpu(s): 2.6% us, 95.8% sy, 0.2% ni, 0.0% id, 0.0% wa, 0.7% hi, 0.7% si Mem:510208k total, 505184k used, 5024k free, 776k buffers Swap: 1959888k total, 2660k used, 1957228k free, 193908k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 2168 root15 -5 0 0 0 D 91.9 0.0 0:11.10 ent:hda8! 10470 benni 18 0 6564 580 432R 3.8 0.10:01.25 cp 4511 root5 -10 100m 55m 6452 S 3.311.2 13:40.66 XFree86 6443 azureus 3519 381m 80m 21m R 0.516.29:57.66 java without azureus, decoding a encrypted file, one second freezed top - 19:43:22 up 8:40, 9 users, load average: 1.89, 1.80, 1.44 Tasks: 110 total, 2 running, 107 sleeping, 0 stopped, 1 zombie Cpu(s): 8.7% us, 89.7% sy, 0.3% ni, 0.0% id, 0.0% wa, 0.7% hi, 0.7% si Mem:510208k total, 504368k used, 5840k free, 7728k buffers Swap: 1959888k total, 2660k used, 1957228k free, 236652k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 10746 benni 25 0 7148 1048 836 D 76.2 0.20:28.56 otrdecoder 2168 root11 -5 0 00 S 19.6 0.0 0:13.63 ent:hda8. 10747 benni 20 0 7148 1048 836 R 1.3 0.20:01.14 otrdecoder 4511 root 5 -10 100m 52m 6452 S 0.7 10.6 13:58.77 XFree86 On Wed, 04 Apr 2007 03:08:36 +0400 Edward Shishkin [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] wrote: Hello. Hello, I have a 120Gb partition with reiser4. If I start a command like cp 500mb file dd within it the system frequently freezes for 1-2 seconds and ent:hda8! uses nearly 100% cpu. This causes the sound output from xmms to be interrupted and my mouse freezes also. If i renice X and xmms to -10 it is okay. Is this the desired behaviour ? No. Can I do sth. else than renicing all time critical programs ? My system is a debian amd64 with linux-2.6.16-mm2. How much RAM is available in your system? If this is much more then 512M, then boot with mem=512M, and let us know if this freeze still takes place (to make sure this is because of unwisely assigned atom_max_size). Thanks, Edward. Best Regards Benni
Re: reiser4 causes system freeze for 1-2 seconds
Hi! Azureus is famous for its fsync() behavior, which is known extremely slow in reiser4. Please search the mailing-list for previous posts. Thanks! 2007/4/5, [EMAIL PROTECTED] [EMAIL PROTECTED]: After some hours the problem has occured again. with azureus running, almost three second freezed top - 19:28:24 up 8:25, 9 users, load average: 6.89, 2.72, 1.26 Tasks: 110 total, 11 running, 98 sleeping, 0 stopped, 1 zombie Cpu(s): 2.6% us, 95.8% sy, 0.2% ni, 0.0% id, 0.0% wa, 0.7% hi, 0.7% si Mem:510208k total, 505184k used, 5024k free, 776k buffers Swap: 1959888k total, 2660k used, 1957228k free, 193908k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 2168 root15 -5 0 0 0 D 91.9 0.0 0:11.10 ent:hda8! 10470 benni 18 0 6564 580 432R 3.8 0.10:01.25 cp 4511 root5 -10 100m 55m 6452 S 3.311.2 13:40.66 XFree86 6443 azureus 3519 381m 80m 21m R 0.516.29:57.66 java without azureus, decoding a encrypted file, one second freezed top - 19:43:22 up 8:40, 9 users, load average: 1.89, 1.80, 1.44 Tasks: 110 total, 2 running, 107 sleeping, 0 stopped, 1 zombie Cpu(s): 8.7% us, 89.7% sy, 0.3% ni, 0.0% id, 0.0% wa, 0.7% hi, 0.7% si Mem:510208k total, 504368k used, 5840k free, 7728k buffers Swap: 1959888k total, 2660k used, 1957228k free, 236652k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 10746 benni 25 0 7148 1048 836 D 76.2 0.20:28.56 otrdecoder 2168 root11 -5 0 00 S 19.6 0.0 0:13.63 ent:hda8. 10747 benni 20 0 7148 1048 836 R 1.3 0.20:01.14 otrdecoder 4511 root 5 -10 100m 52m 6452 S 0.7 10.6 13:58.77 XFree86 On Wed, 04 Apr 2007 03:08:36 +0400 Edward Shishkin [EMAIL PROTECTED] wrote: [EMAIL PROTECTED] wrote: Hello. Hello, I have a 120Gb partition with reiser4. If I start a command like cp 500mb file dd within it the system frequently freezes for 1-2 seconds and ent:hda8! uses nearly 100% cpu. This causes the sound output from xmms to be interrupted and my mouse freezes also. If i renice X and xmms to -10 it is okay. Is this the desired behaviour ? No. Can I do sth. else than renicing all time critical programs ? My system is a debian amd64 with linux-2.6.16-mm2. How much RAM is available in your system? If this is much more then 512M, then boot with mem=512M, and let us know if this freeze still takes place (to make sure this is because of unwisely assigned atom_max_size). Thanks, Edward. Best Regards Benni
Re: reiser4 causes system freeze for 1-2 seconds
[EMAIL PROTECTED] wrote: Hello. Hello, I have a 120Gb partition with reiser4. If I start a command like cp 500mb file dd within it the system frequently freezes for 1-2 seconds and ent:hda8! uses nearly 100% cpu. This causes the sound output from xmms to be interrupted and my mouse freezes also. If i renice X and xmms to -10 it is okay. Is this the desired behaviour ? No. Can I do sth. else than renicing all time critical programs ? My system is a debian amd64 with linux-2.6.16-mm2. How much RAM is available in your system? If this is much more then 512M, then boot with mem=512M, and let us know if this freeze still takes place (to make sure this is because of unwisely assigned atom_max_size). Thanks, Edward. Best Regards Benni
Re: reiser4: reserved disk space
Hello. This is a comment from block_alloc.c: /* * SPACE RESERVED FOR UNLINK/TRUNCATE * * Unlink and truncate require space in transaction (to update stat data, at * least). But we don't want rm(1) to fail with No space on device error. * * Solution is to reserve 5% of disk space for truncates and * unlinks. Specifically, normal space grabbing requests don't grab space from * reserved area. Only requests with BA_RESERVED bit in flags are allowed to * drain it. Per super block delete mutex is used to allow only one * thread at a time to grab from reserved area. * * Grabbing from reserved area should always be performed with BA_CAN_COMMIT * flag. * */ Ignatich wrote: Reiser4 reserves roughly 5% of partition size for it's internal use. This space can be tens of gigabytes for modern hard disks, and unlike ext2 reserve cannot be used even by root in any way. use compression ( http://dev.namesys.com/GettingStartedCryptcompress ) and you will save more then 5% of partition size I changed reserved block count in super.c to be 1% instead of 5%, formatted a test drive in reiser4 and stress tested it by copying many small or few huge files to almost full disk. Everything was working fine. you will encounter problems when using serious workload It there a reason for this reserve to be so big? this value is empirical: it is impossible to estimate this precisely Thanks.
Re: reiser4: reserved disk space
Hello Edward, Thank you for explanation. I'm still not convinced though. Cryptcompress is indeed awesome, but that doesn't change the fact that reserved block calculation can be improved. 5% of modern 500GB drive is 25GB and I can't see how that space will ever be used even if I will delete thousands of files. There is no point in reserving disk space if it will never be used. There are many ways to improve this situation: a upper and lower limit for reserved block count can be used, ability to manually set reserve can be added to mkfs or perhaps reserve usage patterns for various workloads can be analyzed and smarter algorithm can be developed. Can you suggest how to simulate workload that will expose reserve block starvation? Looking forward to your reply, Max Yudin
Re: reiser4 panic in do_readpage_extent
Devils-Hawk wrote:
The problem still persists also trying to boot multiple times it
sometimes triggers much earlier in the boot process than it did before.
Hmm.. can not reproduce it..
The attached patch (against reiser4-for-2.6[19, 20]) allows to dump stack
and some useful info noted as edward-200X among other boot messages.
Would you please send it.
Thanks,
Edward.
regards devh
Edward Shishkin wrote:
Would you please try the attached patch over reiser4-for-2.6.[19, 20]
Thanks,
Edward.
Handle possible race:
do not proceed uf_readpages_filler if page is already uptodate.
Print debugging info.
Signed-off-by: Edward Shishkin [EMAIL PROTECTED]
---
linux-2.6.20-mm2/fs/reiser4/plugin/file/file.c|4 +++
linux-2.6.20-mm2/fs/reiser4/plugin/item/extent_file_ops.c | 17 ++
2 files changed, 21 insertions(+)
--- linux-2.6.20-mm2/fs/reiser4/plugin/file/file.c.orig
+++ linux-2.6.20-mm2/fs/reiser4/plugin/file/file.c
@@ -1619,6 +1619,10 @@
lock_page(page);
cbk_done = 1;
}
+ if (PageUptodate(page)) {
+ unlock_page(page);
+ return 0;
+ }
ret = zload(rc-coord.node);
if (ret) {
unlock_page(page);
--- linux-2.6.20-mm2/fs/reiser4/plugin/item/extent_file_ops.c.orig
+++ linux-2.6.20-mm2/fs/reiser4/plugin/item/extent_file_ops.c
@@ -1157,7 +1157,24 @@
case UNALLOCATED_EXTENT:
j = jfind(mapping, index);
+ if (j == NULL) {
+ dump_stack();
+ printk(edward-2000: oid = %llu\n,
+ (unsigned long long)oid);
+ printk(edward-2001: Jnode not found\n);
+ }
assert(nikita-2688, j);
+ if (jnode_page(j) != NULL) {
+ dump_stack();
+ printk(edward-2002: oid = %llu\n,
+ (unsigned long long)oid);
+ printk(edward-2003: read page %p of idx %lu\n,
+ page, page-index);
+ printk(edward-2004: jnode page %p of idx %lu\n,
+ jnode_page(j), jnode_page(j)-index);
+ printk(edward-2005: jnode blknr = %llu\n,
+ (unsigned long long)(*jnode_get_block(j)));
+ }
assert(vs-1426, jnode_page(j) == NULL);
spin_lock_jnode(j);
Re: reiser4 panic
Hello Matheus, Unfortunately, there is no suggestions except checking this by fsck. Thanks, Edward. Matheus Izvekov wrote: Got this oops message while using reiser4: reiser4[q(3866)]: cbk_level_lookup (fs/reiser4/search.c:961)[vs-3533]: WARNING: Keys are inconsistent. Fsck? reiser4[q(3866)]: cbk_level_lookup (fs/reiser4/search.c:961)[vs-3533]: WARNING: Keys are inconsistent. Fsck? reiser4[q(3866)]: cbk_level_lookup (fs/reiser4/search.c:961)[vs-3533]: WARNING: Keys are inconsistent. Fsck? reiser4[q(3866)]: cbk_level_lookup (fs/reiser4/search.c:961)[vs-3533]: WARNING: Keys are inconsistent. Fsck? reiser4[q(3866)]: cbk_level_lookup (fs/reiser4/search.c:961)[vs-3533]: WARNING: Keys are inconsistent. Fsck? reiser4[q(3866)]: cbk_level_lookup (fs/reiser4/search.c:961)[vs-3533]: WARNING: Keys are inconsistent. Fsck? reiser4[q(3866)]: cbk_level_lookup (fs/reiser4/search.c:961)[vs-3533]: WARNING: Keys are inconsistent. Fsck? reiser4[q(3866)]: cbk_level_lookup (fs/reiser4/search.c:961)[vs-3533]: WARNING: Keys are inconsistent. Fsck? reiser4[q(3866)]: cbk_level_lookup (fs/reiser4/search.c:961)[vs-3533]: WARNING: Keys are inconsistent. Fsck? reiser4[q(3866)]: cbk_level_lookup (fs/reiser4/search.c:961)[vs-3533]: WARNING: Keys are inconsistent. Fsck? reiser4[q(3866)]: cbk_level_lookup (fs/reiser4/search.c:961)[vs-3533]: WARNING: Keys are inconsistent. Fsck? reiser4[q(3866)]: cbk_level_lookup (fs/reiser4/search.c:961)[vs-3533]: WARNING: Keys are inconsistent. Fsck? reiser4[q(3866)]: cbk_level_lookup (fs/reiser4/search.c:961)[vs-3533]: WARNING: Keys are inconsistent. Fsck? reiser4[q(3866)]: cbk_level_lookup (fs/reiser4/search.c:961)[vs-3533]: WARNING: Keys are inconsistent. Fsck? reiser4[q(3866)]: cbk_level_lookup (fs/reiser4/search.c:961)[vs-3533]: WARNING: Keys are inconsistent. Fsck? reiser4[update-eix(3889)]: key_warning (fs/reiser4/plugin/file_plugin_common.c:513)[nikita-717]: WARNING: Error for inode 315366 (-2) reiser4[emerge(4574)]: plugin_by_unsafe_id (fs/reiser4/plugin/plugin.c:276)[nikita-2913]: WARNING: Invalid plugin id: [2:60484] BUG: unable to handle kernel NULL pointer dereference at virtual address 0004 printing eip: f9c9a970 *pde = Oops: [#1] PREEMPT Modules linked in: ntfs bridge ipx p8022 psnap llc p8023 udf isofs zlib_inflate snd_rtctimer pppoe pppox ppp_generic slhc af_packet w83627hf hwmon_vid hwmon eeprom i2c_isa iptable_raw ipt_MASQUERADE iptable_nat ip_nat ipt_tos xt_CLASSIFY xt_mark iptable_mangle xt_tcpudp ipt_set xt_length xt_limit ipt_REJECT xt_conntrack ip_conntrack nfnetlink ipt_ULOG iptable_filter ip_tables x_tables ip_set_portmap fuse snd_pcm_oss snd_mixer_oss snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device rtc reiser4 ext2 mbcache configfs i2c_dev fan button ip_set loop fbcon crc32 font bitblit softcursor vesafb_tng fb vesafb_thread mousedev joydev eth1394 usb_storage usbhid ff_memless nvidia(P) evdev tuner snd_intel8x0 cx8800 snd_ac97_codec sr_mod cx88xx ac97_bus cdrom ir_common i2c_algo_bit snd_pcm video_buf snd_timer psmouse tveeprom compat_ioctl32 btcx_risc videodev snd nvidia_agp i2c_nforce2 serio_raw ohci1394 ieee1394 v4l1_compat ohci_hcd soundcore snd_page_alloc agpgart 8250_pnp 8250 serial_core ehci_hcd uhci_hcd i2c_core pata_amd v4l2_common forcedeth usbcore sg unix CPU:0 EIP:0060:[f9c9a970]Tainted: P VLI EFLAGS: 00210282 (2.6.19.2 #4) EIP is at obtain_item_plugin+0x10/0x20 [reiser4] eax: ebx: d54f0bc4 ecx: c1803040 edx: esi: d54f0bc4 edi: e3888300 ebp: d54f0c5c esp: d54f0b14 ds: 007b es: 007b ss: 0068 Process emerge (pid: 4574, ti=d54f task=efc3faa0 task.ti=d54f) Stack: d54f0bc4 f9c9ab99 d54f0bc4 f9c7caee efc3faa0 d83e7920 f9c5bed2 d83e7920 d83e7920 f9c5b70a e3888300 d54f0bc4 f9cb1ca0 d54f0c5c f9c6fb5f 0050 dc0936c4 0002 d54f0dfc 62696e69 006c6962 Call Trace: [f9c9ab99] item_type_by_coord+0x29/0x30 [reiser4] [f9c7caee] owns_item_common_dir+0x1e/0x80 [reiser4] [f9c5bed2] zparse+0x52/0x70 [reiser4] [f9c5b70a] jload_gfp+0x6a/0x1b0 [reiser4] [f9c6fb5f] coord_by_handle+0x2bf/0xde0 [reiser4] [f9c70928] object_lookup+0xc8/0x110 [reiser4] [f9c879da] find_entry+0xba/0x290 [reiser4] [f9c5ab6f] jrelse+0xf/0x20 [reiser4] [f9c75ab2] init_inode_ordering+0x92/0xa0 [reiser4] [f9c60e7e] longterm_unlock_znode+0x7e/0x1e0 [reiser4] [f9c87c4d] lookup_name+0x9d/0x110 [reiser4] [f9c7a7b5] lookup_common+0x55/0x100 [reiser4] [c017aaad] d_alloc+0x1d/0x1e0 [c016fd98] do_lookup+0x148/0x190 [c0171f5e] __link_path_walk+0x7ee/0xfc0 [c0134cf8] slice+0x8/0x20 [c0134d63] staircase_normal_prio+0x53/0x90 [c017277f] link_path_walk+0x4f/0xe0 [c0115c3a] do_page_fault+0x4ca/0x610 [c0172a1a] do_path_lookup+0xaa/0x280 [c01714db] getname+0x9b/0xf0 [c017347b] __user_walk_fd+0x3b/0x60 [c0166ec5] sys_faccessat+0x95/0x160 [c0115c3a] do_page_fault+0x4ca/0x610 [c0166faf] sys_access+0x1f/0x30 [c0102fa1] sysenter_past_esp+0x56/0x79 === Code: 03 42 24 c3 8d b4 26 00 00
Re: reiser4 panic
On 3/11/07, Edward Shishkin [EMAIL PROTECTED] wrote: Hello Matheus, Unfortunately, there is no suggestions except checking this by fsck. Thanks, Edward. Just did it, here is what i got: FSCK: Node (3112), item (12), [123c7:1(SD):12e747561726567:5a379:0]: item has the wrong length (56). Should be (2). FSCK: Node (3112), item (12), [123c7:1(SD):12e747561726567:5a379:0]: broken item found. FSCK: Node (3112): the node is broken. Pointed from the node (14624), item (47), unit (0). The whole subtree is skipped. FSCK: Node (4294), item (0): Offset (12771) is wrong. Should be (28). FSCK: Node (4294): the node is broken. Pointed from the node (24403), item (55), unit (0). The whole subtree is skipped. FSCK: Node (24453), item (0): Offset (0) is wrong. Should be (28). FSCK: Node (24453): the node is broken. Pointed from the node (24411), item (22), unit (0). The whole subtree is skipped. FSCK: Node (30345), items (13) and (14): Wrong order of keys. FSCK: Node (30345): the node is broken. Pointed from the node (68437), item (26), unit (0). The whole subtree is skipped. FSCK: Node (58606), item (0): Offset (6946) is wrong. Should be (28). FSCK: Node (58606): the node is broken. Pointed from the node (15337), item (39), unit (0). The whole subtree is skipped. FSCK: Node (17165), item (3), unit (1), [1cc66:0(NAME):0:0:0]: unit offset (29281) is wrong. FSCK: Node (17165), item (3), unit (1), [1cc66:0(NAME):0:0:0]: unit offset (29281) is wrong, should be (104). FSCK: Node (17165), item (3), [1cc66:0(NAME):0:0:0]: broken item found. FSCK: Node (17165): the node is broken. Pointed from the node (72100), item (32), unit (0). The whole subtree is skipped. FSCK: Node (20681), item (3), [1d328:1(SD):2e4d616e696665:3f2cf:0]: item has the wrong length (56). Should be (2). FSCK: Node (20681), item (3), [1d328:1(SD):2e4d616e696665:3f2cf:0]: broken item found. FSCK: Node (20681): the node is broken. Pointed from the node (72105), item (25), unit (0). The whole subtree is skipped. FSCK: Node (27567), item (0): Offset (3617) is wrong. Should be (28). FSCK: Node (27567): the node is broken. Pointed from the node (22954), item (31), unit (0). The whole subtree is skipped. FSCK: Node (41824), item (2), unit (29), [30bcf:0(NAME):0:0:0]: unit offset (7738) is wrong. FSCK: Node (41824), item (2), unit (29), [30bcf:0(NAME):0:0:0]: unit offset (7738) is wrong, should be (1868). FSCK: Node (41824), item (2), [30bcf:0(NAME):0:0:0]: broken item found. FSCK: Node (41824): the node is broken. Pointed from the node (71189), item (4), unit (0). The whole subtree is skipped. FSCK: Node (65820), item (0): Offset (14806) is wrong. Should be (28). FSCK: Node (65820): the node is broken. Pointed from the node (71421), item (29), unit (0). The whole subtree is skipped. FSCK: Node (44979), item (0): Offset (15083) is wrong. Should be (28). FSCK: Node (44979): the node is broken. Pointed from the node (72037), item (49), unit (0). The whole subtree is skipped. FSCK: Node (55661), item (14), [44215:0(NAME):0:0:0]: unit count (25185) is not correct. Should be (3). FSCK: Node (55661), item (14), [44215:0(NAME):0:0:0]: entries [0..0] look corrupted. FSCK: Node (55661), item (14), [44215:0(NAME):0:0:0]: broken item found. FSCK: Node (55661): the node is broken. Pointed from the node (72529), item (7), unit (0). The whole subtree is skipped. FSCK: Node (64366): Region of items [9-10] with wrong offsets should be removed. FSCK: Node (64366): Region of items [9-11] with wrong offsets should be removed. FSCK: Node (64366): Region of items [9-12] with wrong offsets should be removed. FSCK: Node (64366): Region of items [9-13] with wrong offsets should be removed. FSCK: Node (64366): Region of items [9-14] with wrong offsets should be removed. FSCK: Node (64366): Region of items [9-15] with wrong offsets should be removed. FSCK: Node (64366): Region of items [9-16] with wrong offsets should be removed. FSCK: Node (64366): Region of items [9-17] with wrong offsets should be removed. FSCK: Node (64366): Region of items [9-18] with wrong offsets should be removed. FSCK: Node (64366): Region of items [9-19] with wrong offsets should be removed. FSCK: Node (64366): Region of items [9-20] with wrong offsets should be removed. FSCK: Node (64366): Region of items [9-21] with wrong offsets should be removed. FSCK: Node (64366): Region of items [9-22] with wrong offsets should be removed. FSCK: Node (64366): Free space start (3222) is wrong. Should be (1195). FSCK: Node (64366): the node is broken. Pointed from the node (72523), item (20), unit (0). The whole subtree is skipped. Then i ran it with --build-fs: FSCK: Node (3112), item (12), [123c7:1(SD):12e747561726567:5a379:0]: item has the wrong length (56). Should be (2). Fixed. FSCK: Node (4294), item (0): Offset (12771) is wrong. Should be (28). Fixed. FSCK: Node (4294), item (0), [13819:4(FB):65786d616373:17465786d616373:d001381d]: does not look likea valid (sdext_symlink) statdata extension. FSCK: Node
Re: reiser4 panic in do_readpage_extent
Devils-Hawk wrote:
Recently tried switching from 2.6.18 + reiser4-for-2.6.18-r3.patch.gz,
which works perfectly fine to 2.6.19 + reiser4-for-2.6.19-r3.patch.gz
I also tried 2.6.20 laurent riffard's reiser4-for-2.6.20. The last
both die somewhere during init when one of the 2 following asserts fails:
extent_file_ops, Line 1160: assert(nikita-2688),j)
extent_file_ops, Line 1161: assert(vs-1426),jnode_page(j) == NULL )
fs was fsck'ed with reiser4progs-1.0.5
regards DevH
Would you please try the attached patch over reiser4-for-2.6.[19, 20]
Thanks,
Edward.
Handle possible race:
do not proceed uf_readpages_filler if page is already uptodate.
Signed-off-by: Edward Shishkin [EMAIL PROTECTED]
---
linux-2.6.20-mm2/fs/reiser4/plugin/file/file.c |4
1 files changed, 4 insertions(+)
--- linux-2.6.20-mm2/fs/reiser4/plugin/file/file.c.orig
+++ linux-2.6.20-mm2/fs/reiser4/plugin/file/file.c
@@ -1619,6 +1619,10 @@
lock_page(page);
cbk_done = 1;
}
+ if (PageUptodate(page)) {
+ unlock_page(page);
+ return 0;
+ }
ret = zload(rc-coord.node);
if (ret) {
unlock_page(page);
Re: reiser4 panic in do_readpage_extent
The problem still persists also trying to boot multiple times it sometimes triggers much earlier in the boot process than it did before. regards devh Edward Shishkin wrote: Would you please try the attached patch over reiser4-for-2.6.[19, 20] Thanks, Edward.
Re: reiser4 panic in do_readpage_extent
Devils-Hawk wrote: Recently tried switching from 2.6.18 + reiser4-for-2.6.18-r3.patch.gz, which works perfectly fine to 2.6.19 + reiser4-for-2.6.19-r3.patch.gz I also tried 2.6.20 laurent riffard's reiser4-for-2.6.20. The last both die somewhere during init when one of the 2 following asserts fails: extent_file_ops, Line 1160: assert(nikita-2688),j) extent_file_ops, Line 1161: assert(vs-1426),jnode_page(j) == NULL ) It seems, new file_read is not happy. Thanks for the report, we'll take a look. Edward. fs was fsck'ed with reiser4progs-1.0.5 regards DevH
Re: Reiser4 ToDo list page
please take a look at http://pub.namesys.com/Reiser4/ToDo why?
Re: Reiser4 terribly slow
I agree, reiser4 has been very slow for me, and as I've upgraded the kernel, it's gotten, if anything, worse. I'm currently running 2.6.18-mm3. I upgraded (as I have before) in the hope that the generally very slow issue had been fixed in this kernel release, and if I knew that it was fixed in the latest mm, I would upgrade in a heartbeat. I'd like to see this fixed, as it's the only real issue that I have. Is this something that can be fixed in a reasonable time, or am I SOL and it's time to switch back to ext3? Have money problems/Han's arrest stopped development? I would think with a 1Ghz K7, I would have a fast enough processor But I don't. Read/write maxes processor usage. I have a regularly scheduled backup that rsyncs with a ext3 disk on the same machine. It slows the machine to a standstill, processor time is 90%+ system. And that's just reading. Forget burning a CD/DVD in a reasonable time. Although now that I have 1Gig of memory inseat of the 128Mb that I had before, I can CACHE the CD image and write it out in a reasonable timeframe. But accessing the disk is SLOW. I disabled fsync in the kernel, and that helped some, but not that much. I've had Reiser4 as my root file system for a number of years now, but I've just about given up. I lust after the advanced features, encryption and change logging in particular, but without a reasonably fast filesystem, I don't see the point. On Wednesday 24 January 2007 17:57, Xu CanHao wrote: AFAIK, vim fsyncs, azureus fsyncs, and may be many other applications fsyncs but not only databases. Definitly turn off fsync() is a bad idea. I just wonder how bad r4's fsync() performance is. It seems the result is: disabled fsync() r4 is even slower than enabled fsync() ext3.o Is it never be possible to improve that? I found in the mailing-list that Hans talked it for many years. this must be a CRITICAL performance flaw for r4. 2007/1/25, [EMAIL PROTECTED] [EMAIL PROTECTED]: On Wed, 24 Jan 2007 22:05:53 +0800, Xu CanHao said: extremely low performance, i managed to turn off fsync() in fs/reiser4/plugin/file/file.c (nullify the sync_unix_file() function), (Maybe you understand the following, if so, feel free to ignore. I'm mostly making sure the list archives have this note so anybody else tempted to do this will think twice)... Note that this can be *very* dangerous to the health of your database if you implement it blindly without understanding the *full* implications. Basically, applications almost never call fsync() unless they need it for database consistency. A system crash at an inopportune time *will* totally ruin your database.
Re: Reiser4 terribly slow
AFAIK, vim fsyncs, azureus fsyncs, and may be many other applications fsyncs but not only databases. Definitly turn off fsync() is a bad idea. I just wonder how bad r4's fsync() performance is. It seems the result is: disabled fsync() r4 is even slower than enabled fsync() ext3.o Is it never be possible to improve that? I found in the mailing-list that Hans talked it for many years. this must be a CRITICAL performance flaw for r4. 2007/1/25, [EMAIL PROTECTED] [EMAIL PROTECTED]: On Wed, 24 Jan 2007 22:05:53 +0800, Xu CanHao said: extremely low performance, i managed to turn off fsync() in fs/reiser4/plugin/file/file.c (nullify the sync_unix_file() function), (Maybe you understand the following, if so, feel free to ignore. I'm mostly making sure the list archives have this note so anybody else tempted to do this will think twice)... Note that this can be *very* dangerous to the health of your database if you implement it blindly without understanding the *full* implications. Basically, applications almost never call fsync() unless they need it for database consistency. A system crash at an inopportune time *will* totally ruin your database.
Re: reiser4: data recovery after mkfs.reiser4?
Hi, On Wed, Dec 20, 2006 at 03:05:33PM -0700, Quinn Harris wrote: I really doubt there is any solution that would take less than a few hours. I am sure it is possible to recover much of the data but to the best of my knowledge no tool exists that can recover from an abandoned root node (for reiser4). Though I believe recovery in this case would just involve finding the root node (think that is reasonably tractable, but slow) fixing the superblock to point to that and let fsck do its thing. I don't think the root node has a magic number that advertises root, but internal nodes do have a recognizable signature and in principal one could deduce which is the root from a collection of the internal nodes. Note that reiser4 packs lots of data in single nodes. If you create a fresh fs with only a few small files they will reside entirely in the root node, which will be clobbered by a mkfs. There is a very good change that mkfs will clobber a little bit of data, but less than 4K. I am sure a few thousand dollars would buy you a solution. Maybe less. Thanks for your concern! However, with the additional flags Vladimir mentioned, I managed to recover all but two shared library files which could be easily copied from a working system. I already answered Vladimir's posting twice - one thanks, I'll try that and one success report. It just so happens that a LOT of mails I send to the list never gets through, no idea why. Regards, Chris
Re: reiser4: data recovery after mkfs.reiser4?
Hi, I already answered Vladimir's posting twice - one thanks, I'll try that and one success report. It just so happens that a LOT of mails I send to the list never gets through, no idea why. Could someone resend that info to the list? I'm sure others would be interested too. Michael
Re: reiser4: data recovery after mkfs.reiser4?
On Thu, Dec 21, 2006 at 12:35:04PM +0100, Michael Weissenbacher wrote: Could someone resend that info to the list? I'm sure others would be interested too. I just replied I'd try what Vladimir said, and so I did - successfully. But here you go, hoping this one will come through. I even stopped signing my outgoing email to this list, to no avail. For me, fsck.reiser4 -u -O --build-fs $device did the trick. The file system was killed by a scripted mkfs.reiser4 followed by a mount and unmount. Only two files were missing afterwards, while lost+found was globbered with the obvious heap of previously deleted files, most of them being 0 in size. Kind regards, Chris
Re: reiser4: data recovery after mkfs.reiser4?
I really doubt there is any solution that would take less than a few hours. I am sure it is possible to recover much of the data but to the best of my knowledge no tool exists that can recover from an abandoned root node (for reiser4). Though I believe recovery in this case would just involve finding the root node (think that is reasonably tractable, but slow) fixing the superblock to point to that and let fsck do its thing. I don't think the root node has a magic number that advertises root, but internal nodes do have a recognizable signature and in principal one could deduce which is the root from a collection of the internal nodes. Note that reiser4 packs lots of data in single nodes. If you create a fresh fs with only a few small files they will reside entirely in the root node, which will be clobbered by a mkfs. There is a very good change that mkfs will clobber a little bit of data, but less than 4K. I am sure a few thousand dollars would buy you a solution. Maybe less. On Tuesday 19 December 2006 1:20 pm, Christian Trefzer wrote: Hi folks, a buggy script I wrote dared to mkfs a reiser4 partition after failing to properly tar up its contents - not that my life would depend on them, but it would save me a few hours if I could get the stuff back. Other than mkfs.reiser4, mount and unmount, nothing was done to the device ; ) Is there any way to recover most of what was stored on the now nuked fs? TIA, Chris
Re: Reiser4 for 2.6.19
great job :) Em Domingo 03 Dezembro 2006 11:49, Laurent Riffard escreveu: [this is a repost, since half of my previous mails didn't reach reiserfs mailing-list] Hi, There is 10 patches in this series, first one is Reiser4 for 2.6.18 version 3. It's made up from the last Namesys Reiser4 patch for vanilla kernel (http://ftp.namesys.com/pub/reiser4-for-2.6/2.6.18/reiser4-for-2.6.18-3.pat ch.gz), updated to apply cleanly on top of 2.6.19 kernel. One can alternatively apply the original patch from Namesys (reiser4-for-2.6.18-3.patch.gz) on top of 2.6.19 kernel, it will successfully apply with some oddities. The 9 next patches are compile-fixes for 2.6.19 or bug-fixes I picked up from reiserfs mailing-list. Andrew Wade (3): Reiser4: fix use after free in jrelse_tail Reiser4: release d_ref Reiser4: release d_ref (fix) Edward Shishkin (2): Reiser4 for 2.6.18 version 3 reiser4-generic_file_read-fix Laurent Riffard (5): Reiser4: cometics changes in mm/filemap.c. Reiser4: fix calls to kmem_cache_destroy Reiser4: Replace inode.u.generic_ip with inode.i_private Reiser4: inode.i_blksize suppression Reiser4: remove unnecessary config.h includes. The whole series is available as in a single patch: http://laurent.riffard.free.fr/reiser4/reiser4-for-2.6.19.patch.gz. I'm not a filesystem guru, nor I'm affiliated with Namesys. These patches just suit my needs. It works for me and I hope it will help somebody else. ~~ laurent
Re: Reiser4 for 2.6.18
ftp://ftp.namesys.com/pub/reiser4-for-2.6/2.6.18/reiser4-for-2.6.18-2.patch.gz 550 No such file or directory. ftp://ftp.namesys.com/pub/reiser4-for-2.6/2.6.18/ftp://ftp.namesys.com/pub/reiser4-for-2.6/2.6.18/reiser4-for-2.6.18-2.patch.gzis empty. :-( On 11/15/06, Edward Shishkin [EMAIL PROTECTED] wrote: Hello everyone. This is made of 2.6.19-rc5-mm1 plus recent changes: ftp://ftp.namesys.com/pub/reiser4-for-2.6/2.6.18/reiser4-for-2.6.18-2.patch.gz Thanks, Edward.
Re: reiser4 experimental patch
Am Freitag, 10. November 2006 00:39 schrieb [EMAIL PROTECTED]: thanks for answer! :) so, the patch compiles fine (one warning in super_ops.c), the FS boot correctly, but if i execute for exemple startx, kernel panic! i compile reiser4 built in with debug, i will send the error (kernel panic) to the list tomorow because i'm now in my house, and the experinet is on my work computer :) I didn't look closer on your patch, but you shouldn't get a warning in super_ops.c. Which compiler do you use? Can you try the patch, I've made? http://www.stud.tu-ilmenau.de/~johi-in/patch-reiser4-2.6.18.bz2
Re: reiser4 experimental patch
Em Sexta 10 Novembro 2006 09:44, Johannes Hirte escreveu: Am Freitag, 10. November 2006 00:39 schrieb [EMAIL PROTECTED]: thanks for answer! :) so, the patch compiles fine (one warning in super_ops.c), the FS boot correctly, but if i execute for exemple startx, kernel panic! i compile reiser4 built in with debug, i will send the error (kernel panic) to the list tomorow because i'm now in my house, and the experinet is on my work computer :) I didn't look closer on your patch, but you shouldn't get a warning in super_ops.c. Which compiler do you use? Can you try the patch, I've made? http://www.stud.tu-ilmenau.de/~johi-in/patch-reiser4-2.6.18.bz2 Hello, i use gcc 3.4.6 on slamd64 (www.slamd64.com), x86_64. i will try your patch :) i'm on-line now on irc.oftc.net nick smyows #reiser4 and my nootebook is on [EMAIL PROTECTED] ... error from boot today.. reiser4 panicked cowaedly: reiser4[mount(1909)]: check_blocks_bitmap (fs/reiser4/plugin/space/bitmap.c:1268)[zam-623]: assertion failed: reiser4_find_next_zero_bit(bnode_working_data(bnode), end_offset, start_offset) = end_offset Kernel panic - not syncing: reiser4[mount(1909)]: check_blocks_bitmap (fs/reiser4/plugin/space/bitmap.c:1268)[zam-623]: assertion failed: reiser4_find_next_zero_bit(bnode_working_data(bnode), end_offset, start_offset) = end_offset ... yesterday give some error but in /fs/reiser4/context.c line 79 i reboot the computer and works now. i'll compare the super_ops.c :) thaks
Re: reiser4 experimental patch
the diference between my an Johannes Hirte's patch is:
/fs/reiser4/plugins/item/item.h
ssize_t (*write) (struct file *, const char __user *, size_t, loff_t
*pos);
---
int (*write) (struct file *, const char __user *, size_t, loff_t
*pos);
*
/fs/reiser4/super_ops.c
290c290
static int reiser4_statfs(struct dentry *dentry, struct kstatfs *statfs)
---
static int reiser4_statfs(struct super_block *super, struct kstatfs *statfs)
292,293d291
struct super_block *super = dentry-d_sb;
571a570,571
// alterado
575,576c575
void *data,
struct vfsmount *mnt)
---
void *data, struct vfsmount *mnt)
582c581,582
static struct file_system_type reiser4_fs_type = {
---
// alterado
struct file_system_type reiser4_fs_type = {
*
i change my super_ops.c but why you alter te int to ssize_t on item.h?
()'s
Em Sexta 10 Novembro 2006 09:57, Guilherme Covolo escreveu:
Em Sexta 10 Novembro 2006 09:44, Johannes Hirte escreveu:
Am Freitag, 10. November 2006 00:39 schrieb [EMAIL PROTECTED]:
thanks for answer! :)
so, the patch compiles fine (one warning in super_ops.c), the FS boot
correctly, but if i execute for exemple startx, kernel panic! i compile
reiser4 built in with debug, i will send the error (kernel panic) to
the list tomorow because i'm now in my house, and the experinet is on
my work computer :)
I didn't look closer on your patch, but you shouldn't get a warning in
super_ops.c. Which compiler do you use?
Can you try the patch, I've made?
http://www.stud.tu-ilmenau.de/~johi-in/patch-reiser4-2.6.18.bz2
Hello,
i use gcc 3.4.6 on slamd64 (www.slamd64.com), x86_64.
i will try your patch :)
i'm on-line now on irc.oftc.net nick smyows #reiser4 and my nootebook is on
[EMAIL PROTECTED] ...
error from boot today..
reiser4 panicked cowaedly: reiser4[mount(1909)]: check_blocks_bitmap
(fs/reiser4/plugin/space/bitmap.c:1268)[zam-623]: assertion failed:
reiser4_find_next_zero_bit(bnode_working_data(bnode), end_offset,
start_offset) = end_offset
Kernel panic - not syncing: reiser4[mount(1909)]: check_blocks_bitmap
(fs/reiser4/plugin/space/bitmap.c:1268)[zam-623]: assertion failed:
reiser4_find_next_zero_bit(bnode_working_data(bnode), end_offset,
start_offset) = end_offset
...
yesterday give some error but in /fs/reiser4/context.c line 79
i reboot the computer and works now.
i'll compare the super_ops.c :)
thaks
Re: reiser4 experimental patch
On Fri, 10 Nov 2006 10:59:30 -0200, Guilherme Covolo said: the diference between my an Johannes Hirte's patch is: * /fs/reiser4/super_ops.c 290c290 static int reiser4_statfs(struct dentry *dentry, struct kstatfs *statfs) --- static int reiser4_statfs(struct super_block *super, struct kstatfs *statfs diff -c or diff -u please. That way, if some unrelated thing moves the lines up or down 1 or 2, it still applies. Also, it's easier to look at a 'diff -u' and understand what's going on, because you get to see 3-4 lines either side of the changed lines. i change my super_ops.c but why you alter te int to ssize_t on item.h? ssize_t isn't an int on some architectures, it's a 'long'. As a result if you reference a 32 bit value where you should use 64, you'll certainly end up with something unexpected (probably an oops). pgp89gE9Ad8ly.pgp Description: PGP signature
Re: reiser4 experimental patch
i change my patch, now is equals of the Johannes Hirte's patch .. only difference is my patch have the comment /* change */ in the source. run on x86_64 fine! :) my patch and the Johannes Hirte's patch is linked in my site, www.youare.not.br thanks to all! Em Sexta 10 Novembro 2006 12:42, [EMAIL PROTECTED] escreveu: On Fri, 10 Nov 2006 10:59:30 -0200, Guilherme Covolo said: the diference between my an Johannes Hirte's patch is: * /fs/reiser4/super_ops.c 290c290 static int reiser4_statfs(struct dentry *dentry, struct kstatfs *statfs) --- static int reiser4_statfs(struct super_block *super, struct kstatfs *statfs diff -c or diff -u please. That way, if some unrelated thing moves the lines up or down 1 or 2, it still applies. Also, it's easier to look at a 'diff -u' and understand what's going on, because you get to see 3-4 lines either side of the changed lines. i change my super_ops.c but why you alter te int to ssize_t on item.h? ssize_t isn't an int on some architectures, it's a 'long'. As a result if you reference a 32 bit value where you should use 64, you'll certainly end up with something unexpected (probably an oops).
Re: reiser4 experimental patch
On Thu, 09 Nov 2006 17:23:20 -0200, Guilherme Covolo said: hello guys, my experimental patch need modfications on fs/reiser4/context.c i need help ;) You'll have to give us more info than that. What happened? Patch reject? It didn't compile? It didn't modprobe? The resulting kernel didn't boot? The resulting kernel oopsed? Other? pgpL4a0CRF33Z.pgp Description: PGP signature
Re: reiser4 experimental patch
thanks for answer! :) so, the patch compiles fine (one warning in super_ops.c), the FS boot correctly, but if i execute for exemple startx, kernel panic! i compile reiser4 built in with debug, i will send the error (kernel panic) to the list tomorow because i'm now in my house, and the experinet is on my work computer :) i don't have much knowledge in C, kernel programer (i'm noob!), and so sorry my poor english :) but if the comunity works together, i beliave it is posibile! thank you so mutch! On Thu, 09 Nov 2006 17:23:20 -0200, Guilherme Covolo said: hello guys, my experimental patch need modfications on fs/reiser4/context.c i need help ;) You'll have to give us more info than that. What happened? Patch reject? It didn't compile? It didn't modprobe? The resulting kernel didn't boot? The resulting kernel oopsed? Other?
Re: reiser4 cryptcompress test setup: bug
Hi, On Mon, 06 Nov 2006 17:07:16 -0500, Valdis.Kletnieks wrote: On Mon, 06 Nov 2006 19:27:57 GMT, Danny Milosavljevic said: Hi Edward, I finally tried your cryptcompress setup (2.6.18-mm3) and just did the first evil thing I could think of: the reiser4 partition with ccreg40 enabled is /dev/sda1 (/mnt/tmp/) (mkfs'ed, thus empty). [ 372.564358] [c01b6400] ext3_dirty_inode+0x30/0x90 [ 372.564364] [c016b4b9] cp_new_stat64+0xf9/0x110 You might want to check /home sometime - this looks like an ext3 botch rather than a reiserfs botch, unless reiser4 is stomping on something behind ext3's back. Well, I thought so at first, but: /home is reiser4 / is ext3 no other ext3s ~/doc/literatur is on /home (checked via 'df .'). So, well, although it looks like it ext3 is the culprit, it is improbable that it actually was the one triggering first (I never saw that backtrace before, too). Naturally once Edwards says he doesn't need the partition for analyzing, I'll nuke it and try to reproduce it again :) cheers, Danny
Re: Reiser4: Running out of room still causes corruption
Am Dienstag, 10. Oktober 2006 07:48 schrieb Daniel Kasak: Hi all. I'd had more problems with filesystem corruption after running out of space. I was able to reproduce this error and log the oops: Oct 16 01:00:01 Theben cron[22195]: (root) CMD (rm -f /var/spool/cron/lastrun/cron.hourly) Oct 16 01:03:08 Theben reiser4[kio_file(10493)]: extent2tail (fs/reiser4/plugin/file/tail_conversion.c:714)[nikita-2282]: Oct 16 01:03:08 Theben WARNING: Partial conversion of 65795: 3 of 4: -28 Oct 16 01:03:08 Theben reiser4[kio_file(10493)]: release_unix_file (fs/reiser4/plugin/file/file.c:2268)[nikita-3233]: Oct 16 01:03:08 Theben WARNING: Failed (-28) to convert in release_unix_file (65795) Oct 16 01:04:23 Theben Unable to handle kernel NULL pointer dereference at 0048 RIP: Oct 16 01:04:23 Theben [8022e638] truncate_inode_pages_range+0x18/0x2d0 Oct 16 01:04:23 Theben PGD 3f5c1067 PUD 349e5067 PMD 0 Oct 16 01:04:23 Theben Oops: [1] PREEMPT Oct 16 01:04:23 Theben CPU 0 Oct 16 01:04:23 Theben Modules linked in: usblp nvidia sym53c8xx ehci_hcd ohci_hcd uhci_hcd Oct 16 01:04:23 Theben Pid: 24786, comm: konqueror Tainted: P 2.6.18-reiser4 #1 Oct 16 01:04:23 Theben RIP: 0010:[8022e638] [8022e638] truncate_inode_pages_range+0x18/0x2d0 Oct 16 01:04:23 Theben RSP: :8100328f7348 EFLAGS: 00010296 Oct 16 01:04:23 Theben RAX: RBX: 810001585548 RCX: 0001 Oct 16 01:04:23 Theben RDX: 3fff RSI: 3000 RDI: Oct 16 01:04:23 Theben RBP: 81000d2b7440 R08: R09: 81002c41ab38 Oct 16 01:04:23 Theben R10: 0040 R11: 0040 R12: 0003 Oct 16 01:04:23 Theben R13: 0001 R14: R15: Oct 16 01:04:23 Theben FS: 2b530b62f260() GS:807ca000() knlGS:f65f8ba0 Oct 16 01:04:23 Theben CS: 0010 DS: ES: CR0: 8005003b Oct 16 01:04:23 Theben CR2: 0048 CR3: 2e6d5000 CR4: 06e0 Oct 16 01:04:23 Theben Process konqueror (pid: 24786, threadinfo 8100328f6000, task 8100224387e0) Oct 16 01:04:23 Theben Stack: 3000 0003 Oct 16 01:04:23 Theben 000160d2 0046 0046 Oct 16 01:04:23 Theben 81003dcb9090 0292 81003dcb9090 00011210 Oct 16 01:04:23 Theben Call Trace: Oct 16 01:04:23 Theben [803263e8] reiser4_invalidate_pages+0x198/0x240 Oct 16 01:04:23 Theben [8034f0c1] item_body_by_coord_hard+0x11/0x20 Oct 16 01:04:23 Theben [80349ede] extent_size+0x1e/0x60 Oct 16 01:04:23 Theben [80349f94] max_unit_key_extent+0x34/0x60 Oct 16 01:04:23 Theben [8034f096] max_item_key_by_coord+0x36/0x50 Oct 16 01:04:23 Theben [8034a378] kill_hook_extent+0x368/0x490 Oct 16 01:04:23 Theben [8032abe6] reiser4_get_neighbor+0xa6/0x4b0 Oct 16 01:04:23 Theben [8034a010] kill_hook_extent+0x0/0x490 Oct 16 01:04:23 Theben [8033e7e2] call_kill_hooks+0x82/0xa0 Oct 16 01:04:23 Theben [8034f096] max_item_key_by_coord+0x36/0x50 Oct 16 01:04:23 Theben [8033f173] prepare_for_compact+0x4a3/0x7e0 Oct 16 01:04:23 Theben [8033e690] kill_units+0x0/0xa0 Oct 16 01:04:23 Theben [80341060] kill_tail+0x0/0x50 Oct 16 01:04:23 Theben [8033e730] kill_head+0x0/0x30 Oct 16 01:04:23 Theben [803156fd] move_lh_internal+0x13d/0x1a0 Oct 16 01:04:23 Theben [80310470] jload_gfp+0x1c0/0x1f0 Oct 16 01:04:23 Theben [8031245c] lock_carry_node+0x2cc/0x330 Oct 16 01:04:23 Theben [80323f4f] handle_eottl+0x5f/0x790 Oct 16 01:04:23 Theben [8033f5cc] kill_node40+0x3c/0xe0 Oct 16 01:04:23 Theben [803134cd] carry_cut+0x4d/0x60 Oct 16 01:04:23 Theben [80312eea] carry+0xda/0x2b0 Oct 16 01:04:23 Theben [80312594] post_carry+0x54/0xd0 Oct 16 01:04:23 Theben [80317b7f] kill_node_content+0x72f/0x7d0 Oct 16 01:04:23 Theben [80315e8e] longterm_lock_znode+0x3de/0x510 Oct 16 01:04:23 Theben [80325dec] coord_by_handle+0x36c/0x3a0 Oct 16 01:04:23 Theben [80341660] lookup_node40+0x320/0x450 Oct 16 01:04:23 Theben [80310470] jload_gfp+0x1c0/0x1f0 Oct 16 01:04:23 Theben [803185af] cut_tree_worker_common+0x27f/0x370 Oct 16 01:04:23 Theben [8032f0f3] plugin_by_unsafe_id+0x23/0x100 Oct 16 01:04:23 Theben [80318330] cut_tree_worker_common+0x0/0x370 Oct 16 01:04:23 Theben [803164c9] cut_tree_object+0x129/0x220 Oct 16 01:04:23 Theben [8031c01e] znode_make_dirty+0x7e/0xc0 Oct 16 01:04:23 Theben [8031a305] reiser4_grab_space+0x45/0xa0 Oct 16 01:04:23 Theben [8031a5ea] reiser4_grab_reserved+0xaa/0x170 Oct 16 01:04:23 Theben [803340fe] cut_file_items+0x12e/0x1d0 Oct 16 01:04:23 Theben [80335020] update_file_size+0x0/0x90
Re: Reiser4: Running out of room still causes corruption
Hello On Tuesday 17 October 2006 16:42, Johannes Hirte wrote: Am Dienstag, 10. Oktober 2006 07:48 schrieb Daniel Kasak: Hi all. I'd had more problems with filesystem corruption after running out of space. This could be deletion of partially converted file. I will try to simulate this case. I was able to reproduce this error and log the oops: Thanks Oct 16 01:00:01 Theben cron[22195]: (root) CMD (rm -f /var/spool/cron/lastrun/cron.hourly) Oct 16 01:03:08 Theben reiser4[kio_file(10493)]: extent2tail (fs/reiser4/plugin/file/tail_conversion.c:714)[nikita-2282]: Oct 16 01:03:08 Theben WARNING: Partial conversion of 65795: 3 of 4: -28 Oct 16 01:03:08 Theben reiser4[kio_file(10493)]: release_unix_file (fs/reiser4/plugin/file/file.c:2268)[nikita-3233]: Oct 16 01:03:08 Theben WARNING: Failed (-28) to convert in release_unix_file (65795) Oct 16 01:04:23 Theben Unable to handle kernel NULL pointer dereference at 0048 RIP: Oct 16 01:04:23 Theben [8022e638] truncate_inode_pages_range+0x18/0x2d0 Oct 16 01:04:23 Theben PGD 3f5c1067 PUD 349e5067 PMD 0 Oct 16 01:04:23 Theben Oops: [1] PREEMPT Oct 16 01:04:23 Theben CPU 0 Oct 16 01:04:23 Theben Modules linked in: usblp nvidia sym53c8xx ehci_hcd ohci_hcd uhci_hcd Oct 16 01:04:23 Theben Pid: 24786, comm: konqueror Tainted: P 2.6.18-reiser4 #1 Oct 16 01:04:23 Theben RIP: 0010:[8022e638] [8022e638] truncate_inode_pages_range+0x18/0x2d0 Oct 16 01:04:23 Theben RSP: :8100328f7348 EFLAGS: 00010296 Oct 16 01:04:23 Theben RAX: RBX: 810001585548 RCX: 0001 Oct 16 01:04:23 Theben RDX: 3fff RSI: 3000 RDI: Oct 16 01:04:23 Theben RBP: 81000d2b7440 R08: R09: 81002c41ab38 Oct 16 01:04:23 Theben R10: 0040 R11: 0040 R12: 0003 Oct 16 01:04:23 Theben R13: 0001 R14: R15: Oct 16 01:04:23 Theben FS: 2b530b62f260() GS:807ca000() knlGS:f65f8ba0 Oct 16 01:04:23 Theben CS: 0010 DS: ES: CR0: 8005003b Oct 16 01:04:23 Theben CR2: 0048 CR3: 2e6d5000 CR4: 06e0 Oct 16 01:04:23 Theben Process konqueror (pid: 24786, threadinfo 8100328f6000, task 8100224387e0) Oct 16 01:04:23 Theben Stack: 3000 0003 Oct 16 01:04:23 Theben 000160d2 0046 0046 Oct 16 01:04:23 Theben 81003dcb9090 0292 81003dcb9090 00011210 Oct 16 01:04:23 Theben Call Trace: Oct 16 01:04:23 Theben [803263e8] reiser4_invalidate_pages+0x198/0x240 Oct 16 01:04:23 Theben [8034f0c1] item_body_by_coord_hard+0x11/0x20 Oct 16 01:04:23 Theben [80349ede] extent_size+0x1e/0x60 Oct 16 01:04:23 Theben [80349f94] max_unit_key_extent+0x34/0x60 Oct 16 01:04:23 Theben [8034f096] max_item_key_by_coord+0x36/0x50 Oct 16 01:04:23 Theben [8034a378] kill_hook_extent+0x368/0x490 Oct 16 01:04:23 Theben [8032abe6] reiser4_get_neighbor+0xa6/0x4b0 Oct 16 01:04:23 Theben [8034a010] kill_hook_extent+0x0/0x490 Oct 16 01:04:23 Theben [8033e7e2] call_kill_hooks+0x82/0xa0 Oct 16 01:04:23 Theben [8034f096] max_item_key_by_coord+0x36/0x50 Oct 16 01:04:23 Theben [8033f173] prepare_for_compact+0x4a3/0x7e0 Oct 16 01:04:23 Theben [8033e690] kill_units+0x0/0xa0 Oct 16 01:04:23 Theben [80341060] kill_tail+0x0/0x50 Oct 16 01:04:23 Theben [8033e730] kill_head+0x0/0x30 Oct 16 01:04:23 Theben [803156fd] move_lh_internal+0x13d/0x1a0 Oct 16 01:04:23 Theben [80310470] jload_gfp+0x1c0/0x1f0 Oct 16 01:04:23 Theben [8031245c] lock_carry_node+0x2cc/0x330 Oct 16 01:04:23 Theben [80323f4f] handle_eottl+0x5f/0x790 Oct 16 01:04:23 Theben [8033f5cc] kill_node40+0x3c/0xe0 Oct 16 01:04:23 Theben [803134cd] carry_cut+0x4d/0x60 Oct 16 01:04:23 Theben [80312eea] carry+0xda/0x2b0 Oct 16 01:04:23 Theben [80312594] post_carry+0x54/0xd0 Oct 16 01:04:23 Theben [80317b7f] kill_node_content+0x72f/0x7d0 Oct 16 01:04:23 Theben [80315e8e] longterm_lock_znode+0x3de/0x510 Oct 16 01:04:23 Theben [80325dec] coord_by_handle+0x36c/0x3a0 Oct 16 01:04:23 Theben [80341660] lookup_node40+0x320/0x450 Oct 16 01:04:23 Theben [80310470] jload_gfp+0x1c0/0x1f0 Oct 16 01:04:23 Theben [803185af] cut_tree_worker_common+0x27f/0x370 Oct 16 01:04:23 Theben [8032f0f3] plugin_by_unsafe_id+0x23/0x100 Oct 16 01:04:23 Theben [80318330] cut_tree_worker_common+0x0/0x370 Oct 16 01:04:23 Theben [803164c9] cut_tree_object+0x129/0x220 Oct 16 01:04:23 Theben [8031c01e] znode_make_dirty+0x7e/0xc0 Oct 16 01:04:23 Theben
Re: Reiser4: Running out of room still causes corruption
On Tuesday 10 October 2006 18:48, Daniel Kasak wrote: I'd had more problems with filesystem corruption after running out of space. So have I, while using Ktorrent on one occasion and rsync to update a portage tree on the other. Both caused fairly massive filesystem corruption, i.e. 200+ files or directories being described as recoverable errors, yet they produced error messages when fsck.reiser4 was run with --fix. Then 200+ objects turned up in lost+found. I do hope this problem can be fixed because in all other features reiser4 is an _excellent_ file system. If there is anything I, as an experienced user, but not much of a programmer any more, can do please ask. Sorry, but I have not made any records of error messages. I have a spare disk on the end of a USB wire with which I am happy to experiment. -- Christopher Sawtell.
Re: reiser4 resize
Alexey Polyakov wrote:
On 9/20/06, Łukasz Mierzwa [EMAIL PROTECTED] wrote:
It's been proven that flushes are doing much more job then they should.
Not so long ago someone send a trace of block device io accesess during
reiser4 work and someone anylized it and said that some files or parts of
file where written over and over 200 times or so.
Wow. I should go back and read that -- I assume this is being worked on?
a few months old filesystem that had been used often just shows a week
spot in reiser4, while downloading files with azureus with only 64KB of
data per second I got disk lid on almost all the time, swithcing to
rtorrent helped as it does not seem to call fsync ( I think I disabled
fsync in azureus).
Hmm, strange. I am using Azureus, but I don't think it's fsync. I can
try rtorrent, but there are several things I like about Azureus that
nothing else seems to do yet.
But also, Azureus didn't always do this. In fact, I used it for several
months before I started having this problem.
Ah, I see, if bittorrent calls fsync often, it's no wonder that
reiser4 behaves badly. I had to preload libnosync for some of my
programs that do fsync to avoid this.
Way ahead of you. I noticed how much fsync performance sucked when
using vim, and I was sick of waiting 10 seconds every time I hit :w -- a
LOT of stuff can pile up in 2 gigs of disk buffer, and at the time,
Reiser4 fsync effectively just called sync.
I didn't know about libnosync (or it didn't exist yet, or didn't work,
I'm not entirely sure), so I was faced with either patching vim, which
had just been patched to _add_ fsync'ing, not to mention all the other
programs that might fsync too much; patching glibc (huge, I don't
update it often, and I'd have no idea where to start); or patching the
kernel.
I now keep backups, and I maintain and apply the following (STUPID,
DON'T TRY THIS AT HOME) patch to my kernel:
--- linux/fs/buffer.c 2006-08-15 20:40:36.504608696 -0500
+++ linux/fs/buffer.c.new 2006-08-15 20:42:35.877461264 -0500
@@ -366,12 +366,12 @@
asmlinkage long sys_fsync(unsigned int fd)
{
- return __do_fsync(fd, 0);
+ return 0;
}
asmlinkage long sys_fdatasync(unsigned int fd)
{
- return __do_fsync(fd, 1);
+ return 0;
}
/*
Re: reiser4 resize
Alexey Polyakov wrote: On 9/19/06, David Masover [EMAIL PROTECTED] wrote: When I have over a gig of RAM free (not even buffer/cache, but _free_), and am trying to download anything over BitTorrent, even if it's less than 200 megs, the disk thrashes so badly that the system is really only usable for web and email. Even movies will occasionally stall when this is happening, and by occasionally, I mean every minute or so. Do you have this problem on plain vanilla + reiser4? Yes. Well, no. My kernel is: vanilla 2.6.17.13 on amd64 patches: sk98lin 8.36, latest from the manufacturer reiser4-for-2.6.17-3 my own patch that disables fsync and fdatasync external modules, installed via Portage: ALSA 1.0.11 driver, using snd_emu10k1 and all sorts of support stuff (OSS emulation, synth, etc) nvidia driver, 1.0.8762 I've also been having a bit of instability issues, but only very rarely do these seem at all FS-related. I'm overclocked a bit, and I can reliably crash my system by playing Neverball, Doom 3, or Quake 4 for several hours. I strongly suspect this is either my overclocking or the nvidia drivers here. However, I doubt anything I've done beyond vanilla+reiser4 is affecting this disk access issue, and I'm pretty much rock solid when I'm not playing a game. I also have a close-to-identical machine nearby which is not overclocked, same kernel, same modules, everything except the nvidia driver, been rock solid for a year, no performance issues to speak of. The main difference, other than graphics, is that the stable machine is using 21 gigs out of 72, whereas the unstable one (the one that's sluggish for BitTorrent) is using 279 gigs out of 350, and has been up to 320 or 330 at least before I started cleaning things out. So I think we're down to two possibilities: Either an update to Azureus has found a way to sync that I'm not aware of, or this is the behavior someone described where Reiser4 will attempt to find contiguous space to allocate, and continue searching and re-searching the same areas of the disk almost every write. To be honest, I hope it's about syncing, somehow, because I'd much rather believe my disk isn't horrendously fragmented...
Re: reiser4 resize
Dnia Thu, 21 Sep 2006 09:30:09 +0200, David Masover [EMAIL PROTECTED]
napisał:
--- linux/fs/buffer.c 2006-08-15 20:40:36.504608696 -0500
+++ linux/fs/buffer.c.new 2006-08-15 20:42:35.877461264 -0500
@@ -366,12 +366,12 @@
asmlinkage long sys_fsync(unsigned int fd)
{
- return __do_fsync(fd, 0);
+ return 0;
}
asmlinkage long sys_fdatasync(unsigned int fd)
{
- return __do_fsync(fd, 1);
+ return 0;
}
/*
I remember that I played a little with disabling sync in reiser4 sources,
it helped only for amarok (it's uses sqlite for storing statistic data and
it writes to it on song change, sqlite calls sync and it ends up with
writeing to disk instead of playing a song, at least on my fs),
bittorrents clients were generating as lot of disk as previously so I gues
it's more likely coused by data/tree fragmentation.
Just my 0,6374526$ ;)
Re: reiser4 resize
Dnia Thu, 21 Sep 2006 09:30:09 +0200, David Masover [EMAIL PROTECTED] napisał: On 9/20/06, Łukasz Mierzwa [EMAIL PROTECTED] wrote: It's been proven that flushes are doing much more job then they should. Not so long ago someone send a trace of block device io accesess during reiser4 work and someone anylized it and said that some files or parts of file where written over and over 200 times or so. Wow. I should go back and read that -- I assume this is being worked on? Well it was not a file but a block, but the effect is the same. http://marc.theaimsgroup.com/?l=reiserfsm=115488109712570w=2
Re: reiser4 resize
On 9/19/06, David Masover [EMAIL PROTECTED] wrote: When I have over a gig of RAM free (not even buffer/cache, but _free_), and am trying to download anything over BitTorrent, even if it's less than 200 megs, the disk thrashes so badly that the system is really only usable for web and email. Even movies will occasionally stall when this is happening, and by occasionally, I mean every minute or so. Do you have this problem on plain vanilla + reiser4? I think some out-of-tree patches (like adaptive readahead?) may result in such behaviour, independent of file system. With a free gig of RAM reiser4 shouldn't access disk a lot, only during flushes. -- Alexey Polyakov
Re: reiser4 resize
On 9/20/06, Łukasz Mierzwa [EMAIL PROTECTED] wrote: It's been proven that flushes are doing much more job then they should. Not so long ago someone send a trace of block device io accesess during reiser4 work and someone anylized it and said that some files or parts of file where written over and over 200 times or so. Bittottent downloads on a few months old filesystem that had been used often just shows a week spot in reiser4, while downloading files with azureus with only 64KB of data per second I got disk lid on almost all the time, swithcing to rtorrent helped as it does not seem to call fsync ( I think I disabled fsync in azureus). Ah, I see, if bittorrent calls fsync often, it's no wonder that reiser4 behaves badly. I had to preload libnosync for some of my programs that do fsync to avoid this. -- Alexey Polyakov
Re: reiser4 resize
Hello On Tuesday 19 September 2006 05:12, Jack Byer wrote: Short summary: Will a resize program for reiser4 be available within the next six months? Currently nobody works on that. So, I guess it is not very likely that reiser4.resize will be created within next six months. Long explanation: I use a 3ware SATA raid card for the main storage on my home network. I currently have 8 250 GB drives in a raid 5 + hot spare configuration. I chose this card because it allows for online capacity expansion. My plan was to wait until a few generations of hard drives emerged then upgrade them one at a time in cycles. This way my storage will periodically expand without any major downtime. When I first created the filesystem, there was a reiser4 resize program. This is no longer the case. that was not a working program. I've began my next upgrade cycle with the purchase of two 750 GB drives. I plan to buy one each month until all eight are replaced. Now I need to make a decision. Do I backup my raid onto the new drives and reformat my raid with another filesystem (xfs, reiser3, jfs), or do I put these new drives into the array and assume that when the time comes to resize the filesystem that a working program will exist? I think you should change to a filesystem which has resize.
Re: reiser4 resize
Vladimir V. Saveliev wrote: Hello On Tuesday 19 September 2006 05:12, Jack Byer wrote: Short summary: Will a resize program for reiser4 be available within the next six months? Currently nobody works on that. So, I guess it is not very likely that reiser4.resize will be created within next six months. Not even an expand? I know a shrink depends on a working repacker (even an offline one), but I'd think expanding it would be simple enough, so long as there's a big warning of You cannot undo this (can't shrink)! When I first created the filesystem, there was a reiser4 resize program. This is no longer the case. that was not a working program. Yes, I remember that, it was a stub. I think you should change to a filesystem which has resize. Alternately, how much would it cost to implement basic resizefs.reiser4? There are other reasons that make me wish I'd stayed away from reiser4 for awhile. Mainly, right now, I need a repacker, and the system seems to have become absurdly slow when it's fragmented. When I have over a gig of RAM free (not even buffer/cache, but _free_), and am trying to download anything over BitTorrent, even if it's less than 200 megs, the disk thrashes so badly that the system is really only usable for web and email. Even movies will occasionally stall when this is happening, and by occasionally, I mean every minute or so. I believe there was a patch to address the thrashing, so I'm eagerly awaiting 2.6.18, but the lack of a repacker bothers me.
Re: reiser4 resize
that was not a working program. I never looked too far into that issue, because I didn't need it back then. I think you should change to a filesystem which has resize. I guess this means that I'll won't use reiser4 again until 2 TB drives come out and I upgrade. Maybe by then reiser4 will be included in the kernel and files-as-directory will work :)
Re: reiser4: mount -o remount,ro / causes error on reboot
On Sun, 17 Sep 2006 21:45:29 +0300, Jussi Suutari-Jääskö wrote: Peter wrote: On Sun, 10 Sep 2006 17:01:18 +, Peter wrote: this bug was also reported on gentoo wrt the newer baselayout. Indications are it may be a r4 issue, although no one seems to know why! http://bugs.gentoo.org/show_bug.cgi?id=144093 I have the same problem, segfault on boot if the system was shutdown properly. For me the trouble appeared after upgrading to glibc-2.4, there's no problem at all with glibc-2.3.6. On my system (amd64 platform with 64-bit gentoo installation) it doesn't have anything to do with baselayout, just the glibc. The fellow who originally posted the bug does not use r4 nor does he use glibc 2.4. http://bugs.gentoo.org/show_bug.cgi?id=147622 Yet, he has the problem we all are enduring. :( -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: reiser4: mount -o remount,ro / causes error on reboot
Peter wrote: On Sun, 10 Sep 2006 17:01:18 +, Peter wrote: this bug was also reported on gentoo wrt the newer baselayout. Indications are it may be a r4 issue, although no one seems to know why! http://bugs.gentoo.org/show_bug.cgi?id=144093 I have the same problem, segfault on boot if the system was shutdown properly. For me the trouble appeared after upgrading to glibc-2.4, there's no problem at all with glibc-2.3.6. On my system (amd64 platform with 64-bit gentoo installation) it doesn't have anything to do with baselayout, just the glibc.
Re: reiser4: mount -o remount,ro / causes error on reboot
On Sun, 10 Sep 2006 17:01:18 +, Peter wrote: this bug was also reported on gentoo wrt the newer baselayout. Indications are it may be a r4 issue, although no one seems to know why! http://bugs.gentoo.org/show_bug.cgi?id=144093 -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: reiser4: mount -o remount,ro / causes error on reboot
On Tue, 12 Sep 2006 21:48:16 -0500, David Masover wrote: snip... Sorry to report this as an r4 bug, although it's interesting to note that the 1.12.4 baselayout did NOT cause this problem in reiserfs3.6 Mine was doubtlessly a Reiser4 bug, as it resulted in either an oops or a panic, I'm not sure which. I think it was an oops, and after that oops, the disk is inaccessible. Since it's a root FS, this is a problem! The init scripts should not be able to cause this, no matter how buggy they are. That's good to know...I guess :) I haven't done this already, because everything works now, and no one's asked me to yet. I find this curious, don't you? I'm wondering if kernel preemption settings and anticipatory read ahead settings could play a role? Mine are CONFIG_PREEMPT=y and CONFIG_DEFAULT_IOSCHED=CFQ. One other thing of note is that my kernel has Jens Axboe's iosched-rollup-2.6.17.4-2 patch. My reiser4 patch is 2.6.17-3. The iosched patch removed a lot of dma messages in the kernel log (not disk errors). Anyway, whatever changed in my downgraded baselayout, at least it's not causing reiser4 to hiccup. Curious problem in that it only occurs immediately after a shutdown, not after the error and a reboot from a maintenance prompt. -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: reiser4: mount -o remount,ro / causes error on reboot
Hello On Wednesday 13 September 2006 01:10, Peter wrote: On Sun, 10 Sep 2006 17:01:18 +, Peter wrote: all snip... To Vladimir and David: This appears to be a nasty gentoo issue. After perusing the forums and bugzilla, it appears that we are not alone in having difficulties with the baselayout. Nonetheless, as the reporter did, I downgraded baselayout from 1.12.4-r7-r7 to 1.11.15-r3 and the reboot problem I noted went away. It is interesting to note that it may be a C program startstop-daemon.c that may be the culprit. I don't expect much help from the gentoo devs since they won't support reiser4, but thought I would throw this out. Sorry to report this as an r4 bug, although it's interesting to note that the 1.12.4 baselayout did NOT cause this problem in reiserfs3.6 I still think that the problem is in reiser4. When the system fails on boot it usually outputs something which may help to understand the problem. Do you see anything like that on faulty startups? You can use either serial or network console to catch kernel output.
Re: reiser4: mount -o remount,ro / causes error on reboot
On Wed, 13 Sep 2006 14:49:05 +0400, Vladimir V. Saveliev wrote: snip... I still think that the problem is in reiser4. When the system fails on boot it usually outputs something which may help to understand the problem. Do you see anything like that on faulty startups? You can use either serial or network console to catch kernel output. Well, the output is from the gentoo rc script and from the imported functions.sh script. It showed two segfaults. It occured afaikt when the dmcrypt addon is called. It uses the start-stop-daemon program. I tried taking a photo of it, but it was blurry and the flash obscured it. I backtracked all the way from the errant baselayout and until 1.11.15-r3 none of the 1.12 series of baselayouts worked. Too bad I can't get gpm to run otherwise I could capture the output. However, the only output are lines from the script with uninitialized variables -- basically the source of the scripts. No dumps, stack traces, or panics. -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: reiser4: mount -o remount,ro / causes error on reboot
On Wed, 13 Sep 2006 14:49:05 +0400, Vladimir V. Saveliev wrote: all snip. Here is a screen shot I posted along with the bug report on this: http://bugs.gentoo.org/attachment.cgi?id=96874action=view . I am sorry the pic is a little blurred, but I had battery trouble. There are two segfaults that occur, 2399 and 2524 and the text that is printed is from line 390 of rc and line 181 of functions.sh. As you can see, there are no panics or dumps and it appears that for whatever reason, the init scripts just cannot continue. However, the reboot (ctrl-d), the scripts execute fine. As I noted previously, the error occurs in all unmasked 1.12 base layout files. 1.11.15-r3 works fine. -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: reiser4: mount -o remount,ro / causes error on reboot
On Sun, 10 Sep 2006 17:01:18 +, Peter wrote: all snip... To Vladimir and David: This appears to be a nasty gentoo issue. After perusing the forums and bugzilla, it appears that we are not alone in having difficulties with the baselayout. Nonetheless, as the reporter did, I downgraded baselayout from 1.12.4-r7-r7 to 1.11.15-r3 and the reboot problem I noted went away. It is interesting to note that it may be a C program startstop-daemon.c that may be the culprit. I don't expect much help from the gentoo devs since they won't support reiser4, but thought I would throw this out. Sorry to report this as an r4 bug, although it's interesting to note that the 1.12.4 baselayout did NOT cause this problem in reiserfs3.6 HTH -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: reiser4: mount -o remount,ro / causes error on reboot
On Tue, 12 Sep 2006 21:10:08 +, Peter wrote: On Sun, 10 Sep 2006 17:01:18 +, Peter wrote: all snip... To Vladimir and David: This appears to be a nasty gentoo issue. After perusing the forums and bugzilla, it appears that we are not alone in having difficulties with the baselayout. Nonetheless, as the reporter did, I downgraded baselayout from 1.12.4-r7-r7 to 1.11.15-r3 and the reboot problem I noted went away. It is interesting to note that it may be a C program startstop-daemon.c that may be the culprit. I don't expect much help from the gentoo devs since they won't support reiser4, but thought I would throw this out. Sorry to report this as an r4 bug, although it's interesting to note that the 1.12.4 baselayout did NOT cause this problem in reiserfs3.6 HTH http://bugs.gentoo.org/show_bug.cgi?id=147274 I have learned that the new baselayout caches some services during startup. While this does not explain why all works fine on a reboot and not on the first boot, perhaps there is something not being cached properly? -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: reiser4: mount -o remount,ro / causes error on reboot
On Mon, 11 Sep 2006 13:10:54 +0200, Sander Sweers wrote: snip... There was a bug in baselayout which caused partition (except /) not to remount ro properly. The bug number is 131001 [1], is this your problem? Greets Sander 1: http://bugs.gentoo.org/show_bug.cgi?id=131001 Thank you, I read that. My version of baselayout has that fix, but that does not appear to be the problem. I think it has more specifically to do with the way the remount option is affecting the reiser4 fs. And, only / is left when the remount,ro part comes anyway. Everything else is unmounted by that time. -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: reiser4: mount -o remount,ro / causes error on reboot
On Mon, 11 Sep 2006 11:30:39 +0400, Vladimir V. Saveliev wrote: snip... Sorry, I am confused. In the first mail you said: On reboot or after a poweroff, root does not mount properly, and after some modules are loaded, there are segfaults when running init scripts. This looks like you have problems on startup. Would you, please, describe the sequence of operations which leads to the problem with more details. I should have written that it occurs always after a normal shutdown or reboot. On initial startup, the error occurs. Then, after CTRL-D, the system reboots and all is fine. Then, after the day, normal shutdown, then abnormal startup. Some more information, I looked at the output from the final mount -v -n -o remont,ro / command, it appears perfectly normal. However, I am not sure it is working normally! -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: reiser4: mount -o remount,ro / causes error on reboot
On Sun, 10 Sep 2006 17:01:18 +, Peter wrote: Using: gentoo kernel 2.6.17.11 with beyond patchset reiser patch 2.6.17-3 reiser4progs 1.0.5 update... Transferring / to a reiser3 partition removes this problem. Shutdown and startup proceed normally. I am using util-linux-2.12r with gentoo patches -r4. This was updated on 9/4/06. I am thinking I will downgrade to -r3 and see if that removes the problem. -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: reiser4: mount -o remount,ro / causes error on reboot
Peter wrote: Using: gentoo kernel 2.6.17.11 with beyond patchset reiser patch 2.6.17-3 reiser4progs 1.0.5 At the end of the gentoo shutdown script is a short function which remounts / as ro. There's also one in the Gentoo startup script, which attempts to remount / ro, then remount it rw. I commented that out, because it was causing similar problems. I figure if it runs sync when it shuts down, that's good enough. Still, it's an annoying problem, I think it's a kernel oops. Namesys, what kind of information would be helpful?
Re: reiser4: mount -o remount,ro / causes error on reboot
On Sun, 10 Sep 2006 15:12:00 -0500, David Masover wrote: Peter wrote: Using: gentoo kernel 2.6.17.11 with beyond patchset reiser patch 2.6.17-3 reiser4progs 1.0.5 At the end of the gentoo shutdown script is a short function which remounts / as ro. There's also one in the Gentoo startup script, which attempts to remount / ro, then remount it rw. I commented that out, because it was causing similar problems. I figure if it runs sync when it shuts down, that's good enough. The errors I note only occur on shutdown (halt.sh) not startup. Do you think it could be an IDE timing thing similar to what was described on another thread on this ml? What's interesting is that this problem is recent and I am trying to look back and see what system-level packages were updated recently (I just converted to the 2006.1 profile before this occured and that caused a lot of programs to recompile. A week earlier, it was gcc-4.1.1. I know base layout was updated recently). Maybe something in mount changed? The shutdown scripts look the same. Something is left unhinged somewhere. Glad I was not hallucinating! Thanks for confirming for me. Still, it's an annoying problem, I think it's a kernel oops. Namesys, what kind of information would be helpful? Yes, it's annoying and disconcerting at the same time. If it was a kernel oop then, wouldn't it have shown itself earlier? I -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: reiser4 corruption on initial copy
hello On Monday 04 September 2006 18:32, Peter wrote: On Fri, 01 Sep 2006 11:02:58 +, Peter wrote: I recently copied over my / partition from a reiserfs to a reiser4 partition. This may be user error, but I wanted to report it anyway. Booting off a live cd- I did the following. snip... I was able to duplicate the problem. Apparently, the Sabayon live cd does not unmount locally mounted partitions. That should not be a problem if sync is called after copy is completed. It looks like Sabayon live cd even does not care to call sync on reboot. Thus, the reiser4 partitions are unclean. When I boot up to the newly created and copied partition, I get the errors I referred to earlier. By manually umounting the partitions I create or copy from -- including /tmp partitions or /var partitions. I do not get the boot time errors.
Re: reiser4 corruption on initial copy
On Tue, 05 Sep 2006 13:30:54 +0400, Vladimir V. Saveliev wrote: hello On Monday 04 September 2006 18:32, Peter wrote: On Fri, 01 Sep 2006 11:02:58 +, Peter wrote: I recently copied over my / partition from a reiserfs to a reiser4 partition. This may be user error, but I wanted to report it anyway. Booting off a live cd- I did the following. snip... I was able to duplicate the problem. Apparently, the Sabayon live cd does not unmount locally mounted partitions. That should not be a problem if sync is called after copy is completed. It looks like Sabayon live cd even does not care to call sync on reboot. which clearly explains a lot of the headaches I had when creating or moving partitions from w/in it! I'll forward this thread to upstream. -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: reiser4 corruption on initial copy
On Fri, 01 Sep 2006 11:02:58 +, Peter wrote: I recently copied over my / partition from a reiserfs to a reiser4 partition. This may be user error, but I wanted to report it anyway. Booting off a live cd- I did the following. snip... I was able to duplicate the problem. Apparently, the Sabayon live cd does not unmount locally mounted partitions. Thus, the reiser4 partitions are unclean. When I boot up to the newly created and copied partition, I get the errors I referred to earlier. By manually umounting the partitions I create or copy from -- including /tmp partitions or /var partitions. I do not get the boot time errors. HTH -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: reiser4 corruption on initial copy
On Fri, 01 Sep 2006 17:35:29 -0500, David Masover wrote: Peter wrote: 2) I did run badblocks on the dest, and it was clean. 3) I am using the patch from 2.6.17.3 and in my kernel, I have full preempt and cfq scheduling. What about the kernel on the livecd? Anticipatory Voluntary Plus, it is smp, so there are some additional options checked. Should I have preempt=none with reiser4? -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: reiser4 corruption on initial copy
Peter wrote: On Fri, 01 Sep 2006 17:35:29 -0500, David Masover wrote: Peter wrote: 2) I did run badblocks on the dest, and it was clean. 3) I am using the patch from 2.6.17.3 and in my kernel, I have full preempt and cfq scheduling. What about the kernel on the livecd? Anticipatory Voluntary Yes, that should be fine, but I was wondering if it's the same version, if you built it yourself, etc etc. Plus, it is smp, so there are some additional options checked. Should I have preempt=none with reiser4? I'm not sure.
Re: reiser4 corruption on initial copy
On Fri, 01 Sep 2006 11:02:58 +, Peter wrote: # cd source partition # tar --one-file-system -cvf - | tar -C dest -xf - of course I had * for all files :) I also did the same with rsync -a /mnt/dest. I then thought perhaps the livecd did not properly unmount the local filesystems, so I issued the umount command. This took approximately three minutes. I then ran an fsck and it appeared fine. However, again on first boot, the kernel issued a slew of errors, mostly about not being able to access /proc (which was there). Then, again, after a reboot, I was OK except that one file /sbin/uname was bad again and needed to be fixed and replaced. I suppose I will try and see if I can tar up the reiser3 partition and restore it in a separate action. If you require additional information, please let me know. -- Peter + Do not reply to this email, it is a spam trap and not monitored. I can be reached via this list, or via jabber: pete4abw at jabber.org ICQ: 73676357
Re: reiser4 corruption on initial copy
Peter wrote: 2) I did run badblocks on the dest, and it was clean. 3) I am using the patch from 2.6.17.3 and in my kernel, I have full preempt and cfq scheduling. What about the kernel on the livecd?
Re: Reiser4 und LZO compression
But speaking of single threadedness, more and more desktops are shipping with ridiculously more power than people need. Even a gamer really Will the LZO compression code in reiser4 be able to use multi-processor systems? E.g. if I've a Turion-X2 in my laptop will it use 2 threads for compression/decompression making cpu throughput much better than whatthe disk could do? lg Clemens 2006/8/30, Hans Reiser [EMAIL PROTECTED]: Edward Shishkin wrote: (Plain) file is considered as a set of logical clusters (64K by default). Minimal unit occupied in memory by (plain) file is one page. Compressed logical cluster is stored on disk in so-called disk clusters. Disk cluster is a set of special items (aka ctails, or compressed bodies), so that one block can contain (compressed) data of many files and everything is packed tightly on disk. So the compression unit is 64k for purposes of your benchmarks.
Re: Reiser4 und LZO compression
Clemens Eisserer wrote: But speaking of single threadedness, more and more desktops are shipping with ridiculously more power than people need. Even a gamer really Will the LZO compression code in reiser4 be able to use multi-processor systems? E.g. if I've a Turion-X2 in my laptop will it use 2 threads for compression/decompression making cpu throughput much better than whatthe disk could do? Compression is going in flush time and there can be more then one flush thread that processes the same transaction atom. Decompression is going in the context of readpage/readpages. So if you mean per file, then yes for compression and no for decompression. Edward. lg Clemens 2006/8/30, Hans Reiser [EMAIL PROTECTED]: Edward Shishkin wrote: (Plain) file is considered as a set of logical clusters (64K by default). Minimal unit occupied in memory by (plain) file is one page. Compressed logical cluster is stored on disk in so-called disk clusters. Disk cluster is a set of special items (aka ctails, or compressed bodies), so that one block can contain (compressed) data of many files and everything is packed tightly on disk. So the compression unit is 64k for purposes of your benchmarks.
Re: Reiser4 und LZO compression
Hi Edward, Thanks a lot for answering. Compression is going in flush time and there can be more then one flush thread that processes the same transaction atom. Decompression is going in the context of readpage/readpages. So if you mean per file, then yes for compression and no for decompression. So the parallelism is not really explicit, more or less a bit accidental. Are threads in the kernel possible - and if yes how large is the typical workload of stuff which can be decompressed? I guess for several hundered kb using more than one thread could speed things up quite a bit? lg Clemens
Re: Reiser4 und LZO compression
Edward Shishkin wrote: Clemens Eisserer wrote: But speaking of single threadedness, more and more desktops are shipping with ridiculously more power than people need. Even a gamer really Will the LZO compression code in reiser4 be able to use multi-processor systems? E.g. if I've a Turion-X2 in my laptop will it use 2 threads for compression/decompression making cpu throughput much better than whatthe disk could do? Compression is going in flush time and there can be more then one flush thread that processes the same transaction atom. Decompression is going in the context of readpage/readpages. So if you mean per file, then yes for compression and no for decompression. I don't think your explanation above is a good one. If there is more than one process reading a file, then you can have multiple decompressions at one time of the same file, yes? Just because there can be more than one flush thread per file does not mean it is likely there will be. CPU scheduling of compression/decompression is an area that could use work in the future.For now, just understand that what we do is better than doing nothing.;-/ Edward. lg Clemens 2006/8/30, Hans Reiser [EMAIL PROTECTED]: Edward Shishkin wrote: (Plain) file is considered as a set of logical clusters (64K by default). Minimal unit occupied in memory by (plain) file is one page. Compressed logical cluster is stored on disk in so-called disk clusters. Disk cluster is a set of special items (aka ctails, or compressed bodies), so that one block can contain (compressed) data of many files and everything is packed tightly on disk. So the compression unit is 64k for purposes of your benchmarks.
Re: Reiser4 und LZO compression
Hans Reiser wrote: Edward Shishkin wrote: Clemens Eisserer wrote: But speaking of single threadedness, more and more desktops are shipping with ridiculously more power than people need. Even a gamer really Will the LZO compression code in reiser4 be able to use multi-processor systems? E.g. if I've a Turion-X2 in my laptop will it use 2 threads for compression/decompression making cpu throughput much better than whatthe disk could do? Compression is going in flush time and there can be more then one flush thread that processes the same transaction atom. Decompression is going in the context of readpage/readpages. So if you mean per file, then yes for compression and no for decompression. I don't think your explanation above is a good one. If there is more than one process reading a file, then you can have multiple decompressions at one time of the same file, yes? You are almost right. Unless they read the same logical cluster. Just because there can be more than one flush thread per file does not mean it is likely there will be. CPU scheduling of compression/decompression is an area that could use work in the future.For now, just understand that what we do is better than doing nothing.;-/ Edward. lg Clemens 2006/8/30, Hans Reiser [EMAIL PROTECTED]: Edward Shishkin wrote: (Plain) file is considered as a set of logical clusters (64K by default). Minimal unit occupied in memory by (plain) file is one page. Compressed logical cluster is stored on disk in so-called disk clusters. Disk cluster is a set of special items (aka ctails, or compressed bodies), so that one block can contain (compressed) data of many files and everything is packed tightly on disk. So the compression unit is 64k for purposes of your benchmarks.
Re: Reiser4 und LZO compression
Clemens Eisserer wrote: But speaking of single threadedness, more and more desktops are shipping with ridiculously more power than people need. Even a gamer really Will the LZO compression code in reiser4 be able to use multi-processor systems? Good point, but it wasn't what I was talking about. I was talking about the compression happening on one CPU, meaning even if it takes most of the CPU to saturate disk throughput, your other CPU is still 100% available, meaning the typical desktop user won't notice their apps running slower, they'll just notice disk access being faster.
Re: Reiser4 und LZO compression
PFC wrote: Maybe, but Reiser4 is supposed to be a general purpose filesystem talking about its advantages/disadvantages wrt. gaming makes sense, I don't see a lot of gamers using Linux ;) There have to be some. Transgaming seems to still be making a successful business out of making games work out-of-the-box under Wine. While I don't imagine there are as many who attempt gaming on Linux, I'd guess a significant portion of Linux users, if not the majority, are at least casual gamers. Some will have given up on the PC as a gaming platform long a go, tired of its upgrade cycle, crashes, game patches, and install times. These people will have a console for games, probably a PS2 so they can watch DVDs, and use their computer for real work, with as much free software as they can manage. Others will compromise somewhat. I compromise by running the binary nVidia drivers, keeping a Windows partition around sometimes, and enjoying many old games which have released their source recently, and now run under Linux -- as well as a few native Linux games, some Cedega games, and some under straight Wine. Basically, I'll play it on Linux if it works well, otherwise I boot Windows. I'm migrating away from that Windows dependency by making sure all my new game purchases work on Linux. Others will use some or all of the above -- stick to old games, use exclusively stuff that works on Linux (one way or the other), or give up on Linux gaming entirely and use a Windows partition. Anything Linux can do to become more game-friendly is one less reason for gamers to have to compromise. Not all gamers are willing to do that. I know at least two who ultimately decided that, with dual boot, they end up spending most of their time on Windows anyway. These are the people who would use Linux if they didn't have a good reason to use something else, but right now, they do. This is not the fault of the filesystem, but taking the attitude of There aren't many Linux gamers anyway -- that's a self-fulfilling prophecy, gamers WILL leave because of it. Also, as you said, gamers (like many others) reinvent filesystems and generally use the Big Zip File paradigm, which is not that stupid for a read only FS (if you cache all file offsets, reading can be pretty fast). However when you start storing ogg-compressed sound and JPEG images inside a zip file, it starts to stink. I don't like it as a read-only FS, either. Take an MMO -- while most commercial ones load the entire game to disk from install DVDs, there are some smaller ones which only cache the data as you explore the world. Also, even with the bigger ones, the world is always changing with patches, and I've seen patches take several hours to install -- not download, install -- on a 2.4 ghz amd64 with 2 gigs of RAM, on a striped RAID. You can trust me when I say this was mostly disk-bound, which is retarded, because it took less than half an hour to install in the first place. Even simple multiplayer games -- hell, even single-player games can get fairly massive updates relatively often. Half-Life 2 is one example -- they've now added HDR to the engine. In these cases, you still need as fast access as possible to the data (to cut down on load time), and it would be nice to save on space as well, but a zipfile starts to make less sense. And yet, I still see people using _cabinet_ files. Compression at the FS layer, plus efficient storing of small files, makes this much simpler. While you can make the zipfile-fs transparent to a game, even your mapping tools, it's still not efficient, and it's not transparent to your modeling package, Photoshop-alike, audio software, or gcc. But everything understands a filesystem. It depends, you have to consider several distinct scenarios. For instance, on a big Postgres database server, the rule is to have as many spindles as you can. - If you are doing a lot of full table scans (like data mining etc), more spindles means reads can be parallelized ; of course this will mean more data will have to be decompressed. I don't see why more spindles means more data decompressed. If anything, I'd imagine it would be less reads, total, if there's any kind of data locality. But I'll leave this to the database experts, for now. - If you are doing a lot of little transactions (web sites), it means seeks can be distributed around the various disks. In this case compression would be a big win because there is free CPU to use ; Dangerous assumption. Three words: Ruby on Rails. There goes your free CPU. Suddenly, compression makes no sense at all. But then, Ruby makes no sense at all for any serious load, unless you really have that much money to spend, or until the Ruby.NET compiler is finished -- that should speed things up. besides, it would virtually double the RAM cache size. No it wouldn't, not the way Reiser4 does it.
Re: Reiser4 und LZO compression
PFC wrote: Maybe, but Reiser4 is supposed to be a general purpose filesystem talking about its advantages/disadvantages wrt. gaming makes sense, I don't see a lot of gamers using Linux ;) But yes, gaming is what pushes hardware development these days, at least on the desktop. Also, as you said, gamers (like many others) reinvent filesystems and generally use the Big Zip File paradigm, which is not that stupid for a read only FS (if you cache all file offsets, reading can be pretty fast). However when you start storing ogg-compressed sound and JPEG images inside a zip file, it starts to stink. *** Does the CPU power necessary to do the compression cost more or less than another drive? *** It depends, you have to consider several distinct scenarios. For instance, on a big Postgres database server, the rule is to have as many spindles as you can. - If you are doing a lot of full table scans (like data mining etc), more spindles means reads can be parallelized ; of course this will mean more data will have to be decompressed. - If you are doing a lot of little transactions (web sites), it means seeks can be distributed around the various disks. In this case compression would be a big win because there is free CPU to use ; besides, it would virtually double the RAM cache size. You have to ponder cost (in CPU $) of compression versus the cost in virtual RAM saved for caching and the cost in disks not bought. *** Do the two processors have separate caches, and thus being overly fined grained makes you memory transfer bound or? It depends on which dual core system you use ; future systems (like Core) will definitely share cache as this is the best option. *** If we analyze the results of my little compression benchmarks, we find that : - gzip is way too slow. - lzo and lzf are pretty close. LZF is faster than LZO (especially on decompression) but compresses worse. So, when we are disk-bound, LZF will be slower. When we are CPU-bound, LZF will be faster. The differences are not that huge, though, so it might be worthwile to weight this against the respective code cleanliness, of which I have no idea. However my compression benchmarks mean nothing because I'm compressing whole files whereas reiser4 will be compressing little blocks of files. We must therefore evaluate the performance of compressors on little blocks, which is very different from 300 megabytes files. For instance, the setup time of the compressor will be important (wether some huffman table needs to be constructed etc), and the compression ratios will be worse. Let's redo a benchmark then. For that I need to know if a compression block in reiser4 will be either : - a FS block containing several files (ie. a block will contain several small files) - a part of a file (ie. a small file will be 1 block) I think it's the second option, right ? (Plain) file is considered as a set of logical clusters (64K by default). Minimal unit occupied in memory by (plain) file is one page. Compressed logical cluster is stored on disk in so-called disk clusters. Disk cluster is a set of special items (aka ctails, or compressed bodies), so that one block can contain (compressed) data of many files and everything is packed tightly on disk.
Re: Reiser4 und LZO compression
Edward Shishkin wrote: (Plain) file is considered as a set of logical clusters (64K by default). Minimal unit occupied in memory by (plain) file is one page. Compressed logical cluster is stored on disk in so-called disk clusters. Disk cluster is a set of special items (aka ctails, or compressed bodies), so that one block can contain (compressed) data of many files and everything is packed tightly on disk. So the compression unit is 64k for purposes of your benchmarks.
Re: Reiser4 und LZO compression
Nigel Cunningham wrote: Hi. On Tue, 2006-08-29 at 06:05 +0200, Jan Engelhardt wrote: Hmm. LZO is the best compression algorithm for the task as measured by the objectives of good compression effectiveness while still having very low CPU usage (the best of those written and GPL'd, there is a slightly better one which is proprietary and uses more CPU, LZRW if I remember right. The gzip code base uses too much CPU, though I think Edward made I don't think that LZO beats LZF in both speed and compression ratio. LZF is also available under GPL (dual-licensed BSD) and was choosen in favor of LZO for the next generation suspend-to-disk code of the Linux kernel. see: http://www.goof.com/pcg/marc/liblzf.html thanks for the info, we will compare them For Suspend2, we ended up converting the LZF support to a cryptoapi plugin. Is there any chance that you could use cryptoapi modules? We could then have a hope of sharing the support. I am throwing in gzip: would it be meaningful to use that instead? The decoder (inflate.c) is already there. 06:04 shanghai:~/liblzf-1.6 l configure* -rwxr-xr-x 1 jengelh users 154894 Mar 3 2005 configure -rwxr-xr-x 1 jengelh users 26810 Mar 3 2005 configure.bz2 -rw-r--r-- 1 jengelh users 30611 Aug 28 20:32 configure.gz-z9 -rw-r--r-- 1 jengelh users 30693 Aug 28 20:32 configure.gz-z6 -rw-r--r-- 1 jengelh users 53077 Aug 28 20:32 configure.lzf We used gzip when we first implemented compression support, and found it to be far too slow. Even with the fastest compression options, we were only getting a few megabytes per second. Perhaps I did something wrong in configuring it, but there's not that many things to get wrong! All that comes to mind is the speed/quality setting -- the number from 1 to 9. Recently, I backed up someone's hard drive using -1, and I believe I was still able to saturate... the _network_. Definitely try again if you haven't changed this, but I can't imagine I'm the first persson to think of it. From what I remember, gzip -1 wasn't faster than the disk. But at least for (very) repetitive data, I was wrong: eve:~ sanity$ time bash -c 'dd if=/dev/zero of=test bs=10m count=10; sync' 10+0 records in 10+0 records out 104857600 bytes transferred in 3.261990 secs (32145287 bytes/sec) real0m3.746s user0m0.005s sys 0m0.627s eve:~ sanity$ time bash -c 'dd if=/dev/zero bs=10m count=10 | gzip -v1 test; sync' 10+0 records in 10+0 records out 104857600 bytes transferred in 2.404093 secs (43616282 bytes/sec) 99.5% real0m2.558s user0m1.554s sys 0m0.680s eve:~ sanity$ This was on OS X, but I think it's still valid -- this is a slightly older Powerbook, with a 5400 RPM drive, 1.6 ghz G4. -1 is still worlds better than nothing. The backup was over 15 gigs, down to about 6 -- loads of repetitive data, I'm sure, but that's where you win with compression anyway. Well, you use cryptoapi anyway, so it should be easy to just let the user pick a plugin, right?
Re: Reiser4 und LZO compression
Nigel Cunningham wrote: Hi. On Mon, 2006-08-28 at 22:15 +0400, Edward Shishkin wrote: Stefan Traby wrote: On Mon, Aug 28, 2006 at 10:06:46AM -0700, Hans Reiser wrote: Hmm. LZO is the best compression algorithm for the task as measured by the objectives of good compression effectiveness while still having very low CPU usage (the best of those written and GPL'd, there is a slightly better one which is proprietary and uses more CPU, LZRW if I remember right. The gzip code base uses too much CPU, though I think Edward made I don't think that LZO beats LZF in both speed and compression ratio. LZF is also available under GPL (dual-licensed BSD) and was choosen in favor of LZO for the next generation suspend-to-disk code of the Linux kernel. see: http://www.goof.com/pcg/marc/liblzf.html thanks for the info, we will compare them For Suspend2, we ended up converting the LZF support to a cryptoapi plugin. Is there any chance that you could use cryptoapi modules? We could then have a hope of sharing the support. No problems with using crypto-api. Reiser4 bypasses it, because currently it supplies the only compression level, which is fairly bad for compressed file systems. Edward.
Re: Reiser4 und LZO compression
Hi. On Tue, 2006-08-29 at 03:23 -0500, David Masover wrote: Nigel Cunningham wrote: Hi. On Tue, 2006-08-29 at 06:05 +0200, Jan Engelhardt wrote: Hmm. LZO is the best compression algorithm for the task as measured by the objectives of good compression effectiveness while still having very low CPU usage (the best of those written and GPL'd, there is a slightly better one which is proprietary and uses more CPU, LZRW if I remember right. The gzip code base uses too much CPU, though I think Edward made I don't think that LZO beats LZF in both speed and compression ratio. LZF is also available under GPL (dual-licensed BSD) and was choosen in favor of LZO for the next generation suspend-to-disk code of the Linux kernel. see: http://www.goof.com/pcg/marc/liblzf.html thanks for the info, we will compare them For Suspend2, we ended up converting the LZF support to a cryptoapi plugin. Is there any chance that you could use cryptoapi modules? We could then have a hope of sharing the support. I am throwing in gzip: would it be meaningful to use that instead? The decoder (inflate.c) is already there. 06:04 shanghai:~/liblzf-1.6 l configure* -rwxr-xr-x 1 jengelh users 154894 Mar 3 2005 configure -rwxr-xr-x 1 jengelh users 26810 Mar 3 2005 configure.bz2 -rw-r--r-- 1 jengelh users 30611 Aug 28 20:32 configure.gz-z9 -rw-r--r-- 1 jengelh users 30693 Aug 28 20:32 configure.gz-z6 -rw-r--r-- 1 jengelh users 53077 Aug 28 20:32 configure.lzf We used gzip when we first implemented compression support, and found it to be far too slow. Even with the fastest compression options, we were only getting a few megabytes per second. Perhaps I did something wrong in configuring it, but there's not that many things to get wrong! All that comes to mind is the speed/quality setting -- the number from 1 to 9. Recently, I backed up someone's hard drive using -1, and I believe I was still able to saturate... the _network_. Definitely try again if you haven't changed this, but I can't imagine I'm the first persson to think of it. From what I remember, gzip -1 wasn't faster than the disk. But at least for (very) repetitive data, I was wrong: eve:~ sanity$ time bash -c 'dd if=/dev/zero of=test bs=10m count=10; sync' 10+0 records in 10+0 records out 104857600 bytes transferred in 3.261990 secs (32145287 bytes/sec) real0m3.746s user0m0.005s sys 0m0.627s eve:~ sanity$ time bash -c 'dd if=/dev/zero bs=10m count=10 | gzip -v1 test; sync' 10+0 records in 10+0 records out 104857600 bytes transferred in 2.404093 secs (43616282 bytes/sec) 99.5% real0m2.558s user0m1.554s sys 0m0.680s eve:~ sanity$ This was on OS X, but I think it's still valid -- this is a slightly older Powerbook, with a 5400 RPM drive, 1.6 ghz G4. -1 is still worlds better than nothing. The backup was over 15 gigs, down to about 6 -- loads of repetitive data, I'm sure, but that's where you win with compression anyway. Wow. That's a lot better; I guess I did get something wrong in trying to tune deflate. That was pre-cryptoapi though; looking at cryptoapi/deflate.c, I don't see any way of controlling the compression level. Am I missing anything? Well, you use cryptoapi anyway, so it should be easy to just let the user pick a plugin, right? Right. They can already pick deflate if they want to. Regards, Nigel
Re: Reiser4 und LZO compression
On 8/29/06, Nigel Cunningham [EMAIL PROTECTED] wrote: Hi. On Tue, 2006-08-29 at 03:23 -0500, David Masover wrote: Nigel Cunningham wrote: We used gzip when we first implemented compression support, and found it to be far too slow. Even with the fastest compression options, we were only getting a few megabytes per second. Perhaps I did something wrong in configuring it, but there's not that many things to get wrong! All that comes to mind is the speed/quality setting -- the number from 1 to 9. Recently, I backed up someone's hard drive using -1, and I believe I was still able to saturate... the _network_. Definitely try again if you haven't changed this, but I can't imagine I'm the first persson to think of it. From what I remember, gzip -1 wasn't faster than the disk. But at least for (very) repetitive data, I was wrong: eve:~ sanity$ time bash -c 'dd if=/dev/zero of=test bs=10m count=10; sync' 10+0 records in 10+0 records out 104857600 bytes transferred in 3.261990 secs (32145287 bytes/sec) real0m3.746s user0m0.005s sys 0m0.627s eve:~ sanity$ time bash -c 'dd if=/dev/zero bs=10m count=10 | gzip -v1 test; sync' 10+0 records in 10+0 records out 104857600 bytes transferred in 2.404093 secs (43616282 bytes/sec) 99.5% real0m2.558s user0m1.554s sys 0m0.680s eve:~ sanity$ This was on OS X, but I think it's still valid -- this is a slightly older Powerbook, with a 5400 RPM drive, 1.6 ghz G4. -1 is still worlds better than nothing. The backup was over 15 gigs, down to about 6 -- loads of repetitive data, I'm sure, but that's where you win with compression anyway. Wow. That's a lot better; I guess I did get something wrong in trying to tune deflate. That was pre-cryptoapi though; looking at cryptoapi/deflate.c, I don't see any way of controlling the compression level. Am I missing anything? Compressing /dev/zero isn't a great test. The timings are really data-dependant: [EMAIL PROTECTED]:~$ time bash -c 'sudo dd if=/dev/zero bs=8M count=64 | gzip -v1 /dev/null' 64+0 records in 64+0 records out 536870912 bytes (537 MB) copied, 7.60817 seconds, 70.6 MB/s 99.6% real0m7.652s user0m6.581s sys 0m0.701s [EMAIL PROTECTED]:~$ time bash -c 'sudo dd if=/dev/mem bs=8M count=64 | gzip -v1 /dev/null' 64+0 records in 64+0 records out 536870912 bytes (537 MB) copied, 21.5863 seconds, 24.9 MB/s 70.4% real0m21.626s user0m18.763s sys 0m1.762s This is on an AMD64 laptop. Ray
Re: Reiser4 und LZO compression
Nigel Cunningham wrote: Hi. On Tue, 2006-08-29 at 03:23 -0500, David Masover wrote: Nigel Cunningham wrote: Hi. On Tue, 2006-08-29 at 06:05 +0200, Jan Engelhardt wrote: Hmm. LZO is the best compression algorithm for the task as measured by the objectives of good compression effectiveness while still having very low CPU usage (the best of those written and GPL'd, there is a slightly better one which is proprietary and uses more CPU, LZRW if I remember right. The gzip code base uses too much CPU, though I think Edward made I don't think that LZO beats LZF in both speed and compression ratio. LZF is also available under GPL (dual-licensed BSD) and was choosen in favor of LZO for the next generation suspend-to-disk code of the Linux kernel. see: http://www.goof.com/pcg/marc/liblzf.html thanks for the info, we will compare them For Suspend2, we ended up converting the LZF support to a cryptoapi plugin. Is there any chance that you could use cryptoapi modules? We could then have a hope of sharing the support. I am throwing in gzip: would it be meaningful to use that instead? The decoder (inflate.c) is already there. 06:04 shanghai:~/liblzf-1.6 l configure* -rwxr-xr-x 1 jengelh users 154894 Mar 3 2005 configure -rwxr-xr-x 1 jengelh users 26810 Mar 3 2005 configure.bz2 -rw-r--r-- 1 jengelh users 30611 Aug 28 20:32 configure.gz-z9 -rw-r--r-- 1 jengelh users 30693 Aug 28 20:32 configure.gz-z6 -rw-r--r-- 1 jengelh users 53077 Aug 28 20:32 configure.lzf We used gzip when we first implemented compression support, and found it to be far too slow. Even with the fastest compression options, we were only getting a few megabytes per second. Perhaps I did something wrong in configuring it, but there's not that many things to get wrong! All that comes to mind is the speed/quality setting -- the number from 1 to 9. Recently, I backed up someone's hard drive using -1, and I believe I was still able to saturate... the _network_. Definitely try again if you haven't changed this, but I can't imagine I'm the first persson to think of it. From what I remember, gzip -1 wasn't faster than the disk. But at least for (very) repetitive data, I was wrong: eve:~ sanity$ time bash -c 'dd if=/dev/zero of=test bs=10m count=10; sync' 10+0 records in 10+0 records out 104857600 bytes transferred in 3.261990 secs (32145287 bytes/sec) real0m3.746s user0m0.005s sys 0m0.627s eve:~ sanity$ time bash -c 'dd if=/dev/zero bs=10m count=10 | gzip -v1 test; sync' 10+0 records in 10+0 records out 104857600 bytes transferred in 2.404093 secs (43616282 bytes/sec) 99.5% real0m2.558s user0m1.554s sys 0m0.680s eve:~ sanity$ This was on OS X, but I think it's still valid -- this is a slightly older Powerbook, with a 5400 RPM drive, 1.6 ghz G4. -1 is still worlds better than nothing. The backup was over 15 gigs, down to about 6 -- loads of repetitive data, I'm sure, but that's where you win with compression anyway. Wow. That's a lot better; I guess I did get something wrong in trying to tune deflate. That was pre-cryptoapi though; looking at cryptoapi/deflate.c, I don't see any way of controlling the compression level. Am I missing anything? zlib is tunable, not cryptoapi's deflate. look at zlib_deflateInit2() Well, you use cryptoapi anyway, so it should be easy to just let the user pick a plugin, right? Right. They can already pick deflate if they want to. Regards, Nigel
Re: Reiser4 und LZO compression
Would it be, by any chance, possible to tweak the thing so that reiserfs plugins become kernel modules, so that the reiserfs core can be put in the kernel without the plugins slowing down its acceptance ? (and updating plugins without rebooting would be a nice extra) The patch below is so-called reiser4 LZO compression plugin as extracted from 2.6.18-rc4-mm3. I think it is an unauditable piece of shit and thus should not enter mainline. Like lib/inflate.c (and this new code should arguably be in lib/). The problem is that if we clean this up, we've diverged very much from the upstream implementation. So taking in fixes and features from upstream becomes harder and more error-prone. I'd suspect that the maturity of these utilities is such that we could afford to turn them into kernel code in the expectation that any future changes will be small. But it's not a completely simple call. (iirc the inflate code had a buffer overrun a while back, which was found and fixed in the upstream version).
Re: Reiser4 und LZO compression
On Tue, Aug 29, 2006 at 03:45:59PM +0200, PFC wrote: Anyone has a bench for lzf ? It's easy, try something like: wget http://www.goof.com/pcg/marc/data/liblzf-1.6.tar.gz tar zxvpf liblzf-1.6.tar.gz cd liblzf-1.6 configure make Now you have a small lzf binary that you can use for testing: cat bigfile|./lzf bigfile.lzf use ./lzf -d for decompression tests -- ciao - Stefan
Re: Reiser4 und LZO compression
On 8/29/06, PFC [EMAIL PROTECTED] wrote: Anyone has a bench for lzf ? This is on a opteron 1.8GHz box. Everything tested hot cache. Testing on a fairly repetative but real test case (an SQL dump of one of the Wikipedia tables): -rw-rw-r-- 1 gmaxwell gmaxwell 426162134 Jul 20 06:54 ../page.sql $time lzop -c ../page.sql page.sql.lzo real0m8.618s user0m7.800s sys 0m0.808s $time lzop -9c ../page.sql page.sql.lzo-9 real4m45.299s user4m44.474s sys 0m0.712s $time gzip -1 -c ../page.sql page.sql.gz real0m19.292s user0m18.545s sys 0m0.748s $time lzop -d -c ./page.sql.lzo /dev/null real0m3.061s user0m2.836s sys 0m0.224s $time gzip -dc page.sql.gz /dev/null real0m7.199s user0m7.020s sys 0m0.176s $time ./lzf -d page.sql.lzf /dev/null real0m2.398s user0m2.224s sys 0m0.172s -rw-rw-r-- 1 gmaxwell gmaxwell 193853815 Aug 29 10:59 page.sql.gz -rw-rw-r-- 1 gmaxwell gmaxwell 243497298 Aug 29 10:47 page.sql.lzf -rw-rw-r-- 1 gmaxwell gmaxwell 259986955 Jul 20 06:54 page.sql.lzo -rw-rw-r-- 1 gmaxwell gmaxwell 204930904 Jul 20 06:54 page.sql.lzo-9 (decompression of the differing lzo levels is the same speed) None of them really decompress fast enough to keep up with the disks in this system, lzf or lzo wouldn't be a big loss. (Bonnie scores: floodlamp,64G,,,246163,52,145536,35,,,365198,42,781.2,2,16,4540,69,+,+++,2454,31,4807,76,+,+++,2027,36)
Re: Reiser4 und LZO compression
I have made a little openoffice spreadsheet with the results. You can have fun entering stuff and seeing the results. http://peufeu.free.fr/compression.ods Basically, a laptop having the same processor as my PC and a crummy 15 MB/s drive (like most laptop drives) will get a 2.5x speedup using lzf, while using 40% CPU for compression and 15% CPU for decompression. I'd say it's a clear, hge win. A desktop computer with a modern IDE drive doing 50 MB/s will still get nice speedups (1.8x on write, 2.5x on read) but of course, more CPU will be used because of the higher throughput. In this case it is CPU limited on compression and disk limited on decompression. However soon everyone will have dual core monsters so... A big ass RAID will not get much benefit unless : - the buffer cache stores compressed pages, so compression virtually doubles the RAM cache - or the CPU is really fast - or you put one of these neat FPGA modules in a free Opteron socket and upload a soft-hardware LZF in it with a few gigabytes/s throughput ...
Re: Reiser4 und LZO compression
PFC wrote: Would it be, by any chance, possible to tweak the thing so that reiserfs plugins become kernel modules, so that the reiserfs core can be put in the kernel without the plugins slowing down its acceptance ? I don't see what this has to do with cryptoapi plugins -- those are not related to Reiser plugins. As for the plugins slowing down acceptance, it's actually the concept of plugins and the plugin API -- in other words, it's the fact that Reiser4 supports plugins -- that is slowing it down, if anything about plugins is still an issue at all. Making them modules would make it worse. Last I saw, Linus doesn't particularly like the idea of plugins because of a few misconceptions, like the possibility of proprietary (possibly GPL-violating) plugins distributed as modules -- basically, something like what nVidia and ATI do with their video drivers. As it is, a good argument in favor of plugins is that this kind of thing isn't possible -- we often put plugins in quotes because really, it's just a nice abstraction layer. They aren't any more plugins than iptables modules or cryptoapi plugins are. If anything, they're less, because they must be compiled into Reiser4, which means either one huge monolithic Reiser4 module (including all plugins), or everything compiled into the kernel image. (and updating plugins without rebooting would be a nice extra) It probably wouldn't be as nice as you think. Remember, if you're using a certain plugin in your root FS, it's part of the FS, so I don't think you'd be able to remove that plugin any more than you're able to remove reiser4.ko if that's your root FS. You'd have to unmount every FS that uses that plugin. At this point, you don't really gain much -- if you unmount every last Reiser4 filesystem, you can then remove reiser4.ko, recompile it, and load a new one with different plugins enabled. Also, these things would typically be part of a kernel update anyway, meaning a reboot anyway. But suppose you could remove a plugin, what then? What would that mean? Suppose half your files are compressed and you remove cryptocompress -- are those files uncompressed when the plugin goes away? Probably not. The only smart way to handle this that I can think of is to make those files unavailable, which is probably not what you want -- how do you update cryptocompress when the new reiser4_cryptocompress.ko is itself compressed? That may be an acceptable solution for some plugins, but you'd have to be extremely careful which ones you remove. The only safe way I can imagine doing this may not be possible, and if it is, it's extremely hackish -- load the plugin under another module name, so r4_cryptocompress would be r4_cryptocompress_init -- have the module, once loaded, do an atomic switch from the old one to the new one, effectively in-place. But that kind of solution is something I've never seen attempted, and only really heard of in strange environments like Erlang. It would probably require much more engineering than the Reiser team can handle right now, especially with their hands full with inclusion. The patch below is so-called reiser4 LZO compression plugin as extracted from 2.6.18-rc4-mm3. I think it is an unauditable piece of shit and thus should not enter mainline. Like lib/inflate.c (and this new code should arguably be in lib/). The problem is that if we clean this up, we've diverged very much from the upstream implementation. So taking in fixes and features from upstream becomes harder and more error-prone. I'd suspect that the maturity of these utilities is such that we could afford to turn them into kernel code in the expectation that any future changes will be small. But it's not a completely simple call. (iirc the inflate code had a buffer overrun a while back, which was found and fixed in the upstream version).
Re: Reiser4 und LZO compression
PFC wrote: I made a little benchmark on my own PC (Athlon64 3200+ in 64 bit gentoo) http://peufeu.free.fr/compression.html So, gzip could be used on PCs having very fast processors and very slow harddrives, like Core Duo laptops. However, lzo compresses nearly as much and is still a lot faster. I don't see a reason for gzip in a FS application. Anyone has a bench for lzf ? Yes, Edward did equivalent tests, and thus we selected LZO.
Re: Reiser4 und LZO compression
PFC, thanks for giving us some real data. May I post it to the lkml thread? In essence, LZO wins the benchmarks, and the code is hard to read. I guess I have to go with LZO, and encourage people to take a stab at dethroning it. Hans PFC wrote: I have made a little openoffice spreadsheet with the results. You can have fun entering stuff and seeing the results. http://peufeu.free.fr/compression.ods Basically, a laptop having the same processor as my PC and a crummy 15 MB/s drive (like most laptop drives) will get a 2.5x speedup using lzf, while using 40% CPU for compression and 15% CPU for decompression. I'd say it's a clear, hge win. A desktop computer with a modern IDE drive doing 50 MB/s will still get nice speedups (1.8x on write, 2.5x on read) but of course, more CPU will be used because of the higher throughput. In this case it is CPU limited on compression and disk limited on decompression. However soon everyone will have dual core monsters so... A big ass RAID will not get much benefit unless : - the buffer cache stores compressed pages, so compression virtually doubles the RAM cache - or the CPU is really fast - or you put one of these neat FPGA modules in a free Opteron socket and upload a soft-hardware LZF in it with a few gigabytes/s throughput Or you look the sysadmin in the eyes, and say, your file servers have more out of disk space problems than load problems, yes? ...
Re: Reiser4 und LZO compression
On 8/29/06, David Masover [EMAIL PROTECTED] wrote: [snip] Conversely, compression does NOT make sense if: - You spend a lot of time with the CPU busy and the disk idle. - You have more than enough disk space. - Disk space is cheaper than buying enough CPU to handle compression. - You've tried compression, and the CPU requirements slowed you more than you saved in disk access. [snip] It's also not always this simple ... if you have a single threaded workload that doesn't overlap CPU and disk well, (de)compression may be free even if you're still CPU bound a lot as the compression is using cpu cycles which would have been otherwise idle..
Re: Reiser4 und LZO compression
Gregory Maxwell wrote: On 8/29/06, David Masover [EMAIL PROTECTED] wrote: [snip] Conversely, compression does NOT make sense if: - You spend a lot of time with the CPU busy and the disk idle. - You have more than enough disk space. - Disk space is cheaper than buying enough CPU to handle compression. - You've tried compression, and the CPU requirements slowed you more than you saved in disk access. [snip] It's also not always this simple ... if you have a single threaded workload that doesn't overlap CPU and disk well, (de)compression may be free even if you're still CPU bound a lot as the compression is using cpu cycles which would have been otherwise idle.. Isn't that implied, though -- if the CPU is not busy (run top under a 2.6 kernel and you'll see an IO-Wait number), then the first condition isn't satisfied -- CPU is not busy, disk is not idle. But speaking of single threadedness, more and more desktops are shipping with ridiculously more power than people need. Even a gamer really won't benefit that much from having a dual-core system, because multithreading is hard, and games haven't been doing it properly. John Carmack is pretty much the only superstar programmer in video games, and after his first fairly massive attempt to make Quake 3 have two threads (since he'd just gotten a dual-core machine to play with) actually resulted in the game running some 30-40% slower than it did with a single thread. So, for the desktop, compression makes perfect sense. We don't have massive amounts of RAID. If we have newer machines, there's a good chance we'll have one CPU sitting mostly idle while playing games. Short of gaming, there are few desktop applications that will fully utilize even one reasonably fast CPU. The reason gamers buy dual-core systems is they're getting cheap enough to be worth it, and that one core sitting idle is a perfect place to do OS/system work not related to the game -- antivirus, automatic update checks, the inevitable background processes leeching a couple few % off your available CPU. So for the typical new desktop with about 2 ghz of 64-bit processor sitting idle, compression is essentially free.
Re: Reiser4 und LZO compression
David Masover wrote: John Carmack is pretty much the only superstar programmer in video games, and after his first fairly massive attempt to make Quake 3 have two threads (since he'd just gotten a dual-core machine to play with) actually resulted in the game running some 30-40% slower than it did with a single thread. Do the two processors have separate caches, and thus being overly fined grained makes you memory transfer bound or? Two processors tends to create a snappier user experience, in that big CPU processes get throttled nicely. Hans
Re: Reiser4 und LZO compression
Hans Reiser wrote: David Masover wrote: John Carmack is pretty much the only superstar programmer in video games, and after his first fairly massive attempt to make Quake 3 have two threads (since he'd just gotten a dual-core machine to play with) actually resulted in the game running some 30-40% slower than it did with a single thread. Do the two processors have separate caches, and thus being overly fined grained makes you memory transfer bound or? It wasn't anything that intelligent. Let me see if I can find it... Taken from http://techreport.com/etc/2005q3/carmack-quakecon/index.x?pg=1 Graphics accelerators are a great example of parallelism working well, he noted, but game code is not similarly parallelizable. Carmack cited his Quake III Arena engine, whose renderer was multithreaded and achieved up to 40% performance increases on multiprocessor systems, as a good example of where games would have to go. (Q3A's SMP mode was notoriously crash-prone and fragile, working only with certain graphics driver revisions and the like.) Initial returns on multithreading, he projected, will be disappointing. Basically, it's hard enough to split what we currently do onto even 2 CPUs, and it definitely seems like we're about to hit a wall in CPU frequency just as multicore becomes a practical reality, so future CPUs may be measured in how many cores they have, not how fast each core is. There's also a question of what to use the extra power for. From the same presentation: Part of the problem with multithreading, argued Carmack, is knowing how to use the power of additional CPU cores to enhance the game experience. A.I., can be effective when very simple, as some of the first Doom logic was. It was less than a page of code, but players ascribed complex behaviors and motivations to the bad guys. However, more complex A.I. seems hard to improve to the point where it really changes the game. More physics detail, meanwhile, threatens to make games too fragile as interactions in the game world become more complex. So, I humbly predict that Physics cards (so-called PPUs) will fail, and be replaced by ever-increasing numbers of cores, which will, for awhile, be one step ahead of what we can think of to fill them with. Thus, anything useful (like compression) that can be split off into a separate thread is going to be useful for games, and won't hurt performance on future mega-multicore monstrosities. The downside is, most game developers are working on Windows, for which FS compression has always sucked. Thus, they most often implement their own compression, often something horrible, like storing the whole game in CAB or ZIP files, and loading the entire level into RAM before play starts, making load times less relevant for gameplay. Reiser4's cryptocompress would be a marked improvement over that, but it would also not be used in many games.
Re: Reiser4 und LZO compression
Hi. On Tue, 2006-08-29 at 15:38 +0400, Edward Shishkin wrote: Nigel Cunningham wrote: Hi. On Tue, 2006-08-29 at 03:23 -0500, David Masover wrote: Nigel Cunningham wrote: Hi. On Tue, 2006-08-29 at 06:05 +0200, Jan Engelhardt wrote: Hmm. LZO is the best compression algorithm for the task as measured by the objectives of good compression effectiveness while still having very low CPU usage (the best of those written and GPL'd, there is a slightly better one which is proprietary and uses more CPU, LZRW if I remember right. The gzip code base uses too much CPU, though I think Edward made I don't think that LZO beats LZF in both speed and compression ratio. LZF is also available under GPL (dual-licensed BSD) and was choosen in favor of LZO for the next generation suspend-to-disk code of the Linux kernel. see: http://www.goof.com/pcg/marc/liblzf.html thanks for the info, we will compare them For Suspend2, we ended up converting the LZF support to a cryptoapi plugin. Is there any chance that you could use cryptoapi modules? We could then have a hope of sharing the support. I am throwing in gzip: would it be meaningful to use that instead? The decoder (inflate.c) is already there. 06:04 shanghai:~/liblzf-1.6 l configure* -rwxr-xr-x 1 jengelh users 154894 Mar 3 2005 configure -rwxr-xr-x 1 jengelh users 26810 Mar 3 2005 configure.bz2 -rw-r--r-- 1 jengelh users 30611 Aug 28 20:32 configure.gz-z9 -rw-r--r-- 1 jengelh users 30693 Aug 28 20:32 configure.gz-z6 -rw-r--r-- 1 jengelh users 53077 Aug 28 20:32 configure.lzf We used gzip when we first implemented compression support, and found it to be far too slow. Even with the fastest compression options, we were only getting a few megabytes per second. Perhaps I did something wrong in configuring it, but there's not that many things to get wrong! All that comes to mind is the speed/quality setting -- the number from 1 to 9. Recently, I backed up someone's hard drive using -1, and I believe I was still able to saturate... the _network_. Definitely try again if you haven't changed this, but I can't imagine I'm the first persson to think of it. From what I remember, gzip -1 wasn't faster than the disk. But at least for (very) repetitive data, I was wrong: eve:~ sanity$ time bash -c 'dd if=/dev/zero of=test bs=10m count=10; sync' 10+0 records in 10+0 records out 104857600 bytes transferred in 3.261990 secs (32145287 bytes/sec) real0m3.746s user0m0.005s sys 0m0.627s eve:~ sanity$ time bash -c 'dd if=/dev/zero bs=10m count=10 | gzip -v1 test; sync' 10+0 records in 10+0 records out 104857600 bytes transferred in 2.404093 secs (43616282 bytes/sec) 99.5% real0m2.558s user0m1.554s sys 0m0.680s eve:~ sanity$ This was on OS X, but I think it's still valid -- this is a slightly older Powerbook, with a 5400 RPM drive, 1.6 ghz G4. -1 is still worlds better than nothing. The backup was over 15 gigs, down to about 6 -- loads of repetitive data, I'm sure, but that's where you win with compression anyway. Wow. That's a lot better; I guess I did get something wrong in trying to tune deflate. That was pre-cryptoapi though; looking at cryptoapi/deflate.c, I don't see any way of controlling the compression level. Am I missing anything? zlib is tunable, not cryptoapi's deflate. look at zlib_deflateInit2() Ok; thanks. I wasn't mistaken then :) Regards, Nigel
Re: Reiser4 und LZO compression
On 29-Aug-06, at 4:03 PM, David Masover wrote: Hans Reiser wrote: David Masover wrote: John Carmack is pretty much the only superstar programmer in video games, and after his first fairly massive attempt to make Quake 3 have two threads (since he'd just gotten a dual-core machine to play with) actually resulted in the game running some 30-40% slower than it did with a single thread. Do the two processors have separate caches, and thus being overly fined grained makes you memory transfer bound or? It wasn't anything that intelligent. Let me see if I can find it... Taken from http://techreport.com/etc/2005q3/carmack-quakecon/index.x?pg=1 Graphics accelerators are a great example of parallelism working well, he noted, but game code is not similarly parallelizable. ... The downside is, most game developers are working on Windows, for which FS compression has always sucked. Thus, they most often implement their own compression, often something horrible, like storing the whole game in CAB or ZIP files, and loading the entire level into RAM before play starts, making load times less relevant for gameplay. Reiser4's cryptocompress would be a marked improvement over that, but it would also not be used in many games. Gamer systems, whether from coder's or player's p.o.v., would appear fairly irrelevant to reiserfs and this list. I'd trust Carmack's eye candy credentials but doubt he has much to say about filesystems or server threading...
Re: Reiser4 und LZO compression
Toby Thain wrote: Gamer systems, whether from coder's or player's p.o.v., would appear fairly irrelevant to reiserfs and this list. I'd trust Carmack's eye candy credentials but doubt he has much to say about filesystems or server threading... Maybe, but Reiser4 is supposed to be a general purpose filesystem, so talking about its advantages/disadvantages wrt. gaming makes sense, especially considering gamers are the most likely to tune their desktop for perfomance. That was a bit much, though. I apologize.
Re: Reiser4 und LZO compression
On Sun, 27 August 2006 01:04:28 -0700, Andrew Morton wrote: Like lib/inflate.c (and this new code should arguably be in lib/). The problem is that if we clean this up, we've diverged very much from the upstream implementation. So taking in fixes and features from upstream becomes harder and more error-prone. I've had an identical argument with Linus about lib/zlib_*. He decided that he didn't care about diverging, I went ahead and changed the code. In the process, I merged a couple of outstanding bugfixes and reduced memory consumption by 25%. Looks like Linus was right on that one. I'd suspect that the maturity of these utilities is such that we could afford to turn them into kernel code in the expectation that any future changes will be small. But it's not a completely simple call. (iirc the inflate code had a buffer overrun a while back, which was found and fixed in the upstream version). Dito in lib/zlib_*. lib/inflage.c is only used for the various in-kernel bootloaders to uncompress a kernel image. Anyone tampering with the image to cause a buffer overrun already owns the machine anyway. Whether any of our experiences with zlib apply to lzo remains a question, though. Jörn -- I've never met a human being who would want to read 17,000 pages of documentation, and if there was, I'd kill him to get him out of the gene pool. -- Joseph Costello
Re: Reiser4 und LZO compression
Alexey Dobriyan wrote: Reiser4 developers, Andrew, The patch below is so-called reiser4 LZO compression plugin as extracted from 2.6.18-rc4-mm3. I think it is an unauditable piece of shit and thus should not enter mainline. Hmm. LZO is the best compression algorithm for the task as measured by the objectives of good compression effectiveness while still having very low CPU usage (the best of those written and GPL'd, there is a slightly better one which is proprietary and uses more CPU, LZRW if I remember right. The gzip code base uses too much CPU, though I think Edward made an option of it). Could you be kind enough to send me a plugin which is better at those two measures, I'd be quite grateful? By the way, could you tell me about this auditing stuff? Last I remember, when I mentioned that the US Defense community had coding practices worth adopting by the Kernel Community, I was pretty much disregarded. So, while I understand that the FSB has serious security issues what with all these Americans seeking to crack their Linux boxen, complaining to me about auditability seems a bit graceless.;-) Especially if there is no offer of replacement compression code. Oh, and this LZO code is not written by Namesys. You can tell by the utter lack of comments, assertions, etc. We are just seeking to reuse well known widely used code. I have in the past been capable of demanding that my programmers comment code not written by them before we use it, but this time I did not. I have mixed feeling about us adding our comments to code written by a compression specialist. If Andrew wants us to write our own compression code, or comment this code and fill it with asserts, we will grumble a bit and do it. It is not a task I am eager for, as compression code is a highly competitive field which gives me the surface impression that if you are not gripped by what you are sure is an inspiration you should stay out of it. Jorn wrote: I've had an identical argument with Linus about lib/zlib_*. He decided that he didn't care about diverging, I went ahead and changed the code. In the process, I merged a couple of outstanding bugfixes and reduced memory consumption by 25%. Looks like Linus was right on that one. Anyone sends myself or Edward a patch, that's great. Jorn, sounds like you did a good job on that one. Hans
Re: Reiser4 und LZO compression
On Sun, 27 Aug 2006 04:42:59 -0500 David Masover [EMAIL PROTECTED] wrote: Andrew Morton wrote: On Sun, 27 Aug 2006 04:34:26 +0400 Alexey Dobriyan [EMAIL PROTECTED] wrote: The patch below is so-called reiser4 LZO compression plugin as extracted from 2.6.18-rc4-mm3. I think it is an unauditable piece of shit and thus should not enter mainline. Like lib/inflate.c (and this new code should arguably be in lib/). The problem is that if we clean this up, we've diverged very much from the upstream implementation. So taking in fixes and features from upstream becomes harder and more error-prone. Well, what kinds of changes have to happen? I doubt upstream would care about moving some of it to lib/ -- and anyway, reiserfs-list is on the CC. We are speaking of upstream in the third party in the presence of upstream, so... The ifdef jungle is ugly, and especially the WIN / 16-bit DOS stuff is completely useless here. Maybe just ask upstream? I am not sure if Mr. Oberhumer still cares about LZO 1.x, AFAIK he now develops a new compressor under a commercial license. Regards, -- Jindrich Makovicka
Re: Reiser4 und LZO compression
On Mon, Aug 28, 2006 at 10:06:46AM -0700, Hans Reiser wrote: Hmm. LZO is the best compression algorithm for the task as measured by the objectives of good compression effectiveness while still having very low CPU usage (the best of those written and GPL'd, there is a slightly better one which is proprietary and uses more CPU, LZRW if I remember right. The gzip code base uses too much CPU, though I think Edward made I don't think that LZO beats LZF in both speed and compression ratio. LZF is also available under GPL (dual-licensed BSD) and was choosen in favor of LZO for the next generation suspend-to-disk code of the Linux kernel. see: http://www.goof.com/pcg/marc/liblzf.html -- ciao - Stefan
Re: Reiser4 und LZO compression
Jindrich Makovicka wrote: On Sun, 27 Aug 2006 04:42:59 -0500 David Masover [EMAIL PROTECTED] wrote: Andrew Morton wrote: On Sun, 27 Aug 2006 04:34:26 +0400 Alexey Dobriyan [EMAIL PROTECTED] wrote: The patch below is so-called reiser4 LZO compression plugin as extracted from 2.6.18-rc4-mm3. I think it is an unauditable piece of shit and thus should not enter mainline. Like lib/inflate.c (and this new code should arguably be in lib/). The problem is that if we clean this up, we've diverged very much from the upstream implementation. So taking in fixes and features from upstream becomes harder and more error-prone. Well, what kinds of changes have to happen? I doubt upstream would care about moving some of it to lib/ -- and anyway, reiserfs-list is on the CC. We are speaking of upstream in the third party in the presence of upstream, so... The ifdef jungle is ugly, and especially the WIN / 16-bit DOS stuff is completely useless here. I agree that it needs some brushing, putting in todo.. Maybe just ask upstream? I am not sure if Mr. Oberhumer still cares about LZO 1.x, AFAIK he now develops a new compressor under a commercial license. Regards,
Re: Reiser4 und LZO compression
Stefan Traby wrote: On Mon, Aug 28, 2006 at 10:06:46AM -0700, Hans Reiser wrote: Hmm. LZO is the best compression algorithm for the task as measured by the objectives of good compression effectiveness while still having very low CPU usage (the best of those written and GPL'd, there is a slightly better one which is proprietary and uses more CPU, LZRW if I remember right. The gzip code base uses too much CPU, though I think Edward made I don't think that LZO beats LZF in both speed and compression ratio. LZF is also available under GPL (dual-licensed BSD) and was choosen in favor of LZO for the next generation suspend-to-disk code of the Linux kernel. see: http://www.goof.com/pcg/marc/liblzf.html thanks for the info, we will compare them
Re: Reiser4 und LZO compression
Hi. On Mon, 2006-08-28 at 22:15 +0400, Edward Shishkin wrote: Stefan Traby wrote: On Mon, Aug 28, 2006 at 10:06:46AM -0700, Hans Reiser wrote: Hmm. LZO is the best compression algorithm for the task as measured by the objectives of good compression effectiveness while still having very low CPU usage (the best of those written and GPL'd, there is a slightly better one which is proprietary and uses more CPU, LZRW if I remember right. The gzip code base uses too much CPU, though I think Edward made I don't think that LZO beats LZF in both speed and compression ratio. LZF is also available under GPL (dual-licensed BSD) and was choosen in favor of LZO for the next generation suspend-to-disk code of the Linux kernel. see: http://www.goof.com/pcg/marc/liblzf.html thanks for the info, we will compare them For Suspend2, we ended up converting the LZF support to a cryptoapi plugin. Is there any chance that you could use cryptoapi modules? We could then have a hope of sharing the support. Regards, Nigel
Re: Reiser4 und LZO compression
Nigel Cunningham wrote: For Suspend2, we ended up converting the LZF support to a cryptoapi plugin. Is there any chance that you could use cryptoapi modules? We could then have a hope of sharing the support It is in principle a good idea, and I hope we will be able to say yes. However, I have to see the numbers, as we are more performance sensitive than you folks probably are, and every 10% is a big deal for us.
Re: Reiser4 und LZO compression
Hmm. LZO is the best compression algorithm for the task as measured by the objectives of good compression effectiveness while still having very low CPU usage (the best of those written and GPL'd, there is a slightly better one which is proprietary and uses more CPU, LZRW if I remember right. The gzip code base uses too much CPU, though I think Edward made I don't think that LZO beats LZF in both speed and compression ratio. LZF is also available under GPL (dual-licensed BSD) and was choosen in favor of LZO for the next generation suspend-to-disk code of the Linux kernel. see: http://www.goof.com/pcg/marc/liblzf.html thanks for the info, we will compare them For Suspend2, we ended up converting the LZF support to a cryptoapi plugin. Is there any chance that you could use cryptoapi modules? We could then have a hope of sharing the support. I am throwing in gzip: would it be meaningful to use that instead? The decoder (inflate.c) is already there. 06:04 shanghai:~/liblzf-1.6 l configure* -rwxr-xr-x 1 jengelh users 154894 Mar 3 2005 configure -rwxr-xr-x 1 jengelh users 26810 Mar 3 2005 configure.bz2 -rw-r--r-- 1 jengelh users 30611 Aug 28 20:32 configure.gz-z9 -rw-r--r-- 1 jengelh users 30693 Aug 28 20:32 configure.gz-z6 -rw-r--r-- 1 jengelh users 53077 Aug 28 20:32 configure.lzf Jan Engelhardt --
