This is on a dual-CPU opteron system, with 2 x 3ware 9500 12-channel SATA controllers for a total of 8.5TB; I've configured a RAID5 over each 3ware controller, and use linux md RAID0 over those two "devices". There was an issue with linux md RAID0 for that size, but that's been resolved (at least, the problem I had first :-)
The device itself seems to work fine, as reiser4 works. I wanted to compare to reiserfs 3.6, so I created a reiserfs, mounted it, and tried to use it. Running bonnie++ on it caused an oops, apparently in the reiserfs code: kernel: Unable to handle kernel paging request at 00002aaaaaad79ea RIP: kernel: <ffffffff801b0ac1>{scan_bitmap_block+129} kernel: PGD 954ad067 PUD 9a1af067 PMD b75bc067 PTE 0 kernel: Oops: 0000 [1] SMP kernel: CPU 0 kernel: Modules linked in: raid0 reiser4 zlib_deflate zlib_inflate raid5 raid6 xor ipv6 evdev tg3 3w_9xxx hw_random i2c_amd756 i2c_amd8111 i2c_core psmouse rtc kernel: Pid: 12006, comm: bonnie++ Not tainted 2.6.12.2.raid0fixreiser4 kernel: RIP: 0010:[scan_bitmap_block+129/768] <ffffffff801b0ac1>{scan_bitmap_block+129} kernel: RSP: 0018:ffff81007f461a18 EFLAGS: 00010286 kernel: RAX: 00002aaaaaad79ea RBX: 000000000000001c RCX: 00000000000021e0 kernel: RDX: 0000000000000000 RSI: 000000000000001c RDI: ffff81007f461d98 kernel: RBP: ffffc2000050a1c0 R08: 0000000000000001 R09: 0000000000000011 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff81007f461a9c kernel: R13: 00000000000021e0 R14: ffff81007f0ffc00 R15: 0000000000000011 kernel: FS: 00002aaaab26f8c0(0000) GS:ffffffff804b7480(0000) knlGS:0000000000000000 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b kernel: CR2: 00002aaaaaad79ea CR3: 00000000d9d11000 CR4: 00000000000006e0 kernel: Process bonnie++ (pid: 12006, threadinfo ffff81007f460000, task ffff81007feee0f0) kernel: Stack: 0000000000000010 00000000801cabce 0000001c00000001 ffff81007f461d98 kernel: ffff81009f6c9018 000000000000001c ffff81007f0ffc00 000000000000001c kernel: ffff81007f461d98 0000000000000001 kernel: Call Trace:<ffffffff801b10a9>{scan_bitmap+585} <ffffffff801b2313>{reiserfs_allocate_blocknrs+803} kernel: <ffffffff801bc08c>{reiserfs_allocate_blocks_for_region+524} kernel: <ffffffff80178f36>{alloc_page_buffers+102} <ffffffff801ca1a8>{pathrelse+40} kernel: <ffffffff801484b0>{autoremove_wake_function+0} <ffffffff801be315>{reiserfs_file_write+1045} kernel: <ffffffff801655ad>{do_anonymous_page+861} <ffffffff80165b90>{handle_mm_fault+304} kernel: <ffffffff80203dd1>{__up_read+33} <ffffffff8011e289>{do_page_fault+601} kernel: <ffffffff8032a7d3>{_spin_lock+3} <ffffffff80176c90>{vfs_write+192} kernel: <ffffffff80176de3>{sys_write+83} <ffffffff8010d91a>{system_call+126} kernel: kernel: kernel: Code: 8b 00 48 c1 e8 02 a8 01 74 17 49 8b 86 70 02 00 00 48 ff 80 kernel: RIP <ffffffff801b0ac1>{scan_bitmap_block+129} RSP <ffff81007f461a18> kernel: CR2: 00002aaaaaad79ea I rebooted (hard, as a shutdown didn't work...). After that, I tried a mkfs followd by an fsck, which gives an error! Here's the console log: satazilla:~# mkfs.reiserfs /dev/md13 mkfs.reiserfs 3.6.19 (2003 www.namesys.com) A pair of credits: Joshua Macdonald wrote the first draft of the transaction manager. Yuri Rupasov did testing and benchmarking, plus he invented the r5 hash (also used by the dcache code). Yura Rupasov, Anatoly Pinchuk, Igor Krasheninnikov, Grigory Zaigralin, Mikhail Gilula, Igor Zagorovsky, Roman Pozlevich, Konstantin Shvachko, and Joshua MacDonald are former contributors to the project. The Defense Advanced Research Projects Agency (DARPA, www.darpa.mil) is the primary sponsor of Reiser4. DARPA does not endorse this project; it merely sponsors it. Guessing about desired format.. Kernel 2.6.12.2.raid0fixreiser4 is running. Format 3.6 with standard journal Count of blocks on the device: 2148377056 Number of blocks consumed by mkreiserfs formatting process: 8239 Blocksize: 4096 Hash function used to sort names: "r5" Journal Size 8193 blocks (first block 18) Journal Max transaction length 1024 inode generation number: 0 UUID: 10ae60ee-1abb-49d1-ae55-cf238626c0b5 ATTENTION: YOU SHOULD REBOOT AFTER FDISK! ALL DATA WILL BE LOST ON '/dev/md13'! Continue (y/n):y Initializing journal - 0%....20%....40%....60%....80%....100% Syncing..ok Tell your friends to use a kernel based on 2.4.18 or later, and especially not a kernel based on 2.4.9, when you use reiserFS. Have fun. ReiserFS is successfully created on /dev/md13. satazilla:~# reiserfsck /dev/md13 reiserfsck 3.6.19 (2003 www.namesys.com) ************************************************************* ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to reiserfs-list@namesys.com, ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** ************************************************************* Will read-only check consistency of the filesystem on /dev/md13 Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes ########### reiserfsck --check started at Tue Jul 19 13:29:22 2005 ########### Replaying journal.. No transactions found reiserfs_open_ondisk_bitmap: wrong either bitmaps number, count of blocks or blocksize, run with --rebuild-sb to fix it reiserfsck: Could not open bitmap When I try running with --rebuild-sb it says: Reiserfs super block in block 16 on 0x90d of format 3.6 with standard journal Count of blocks on the device: 2148377056 Number of bitmaps: 28 Blocksize: 4096 Free blocks (count of blocks - used [journal, bitmaps, data, reserved] blocks): 2148368817 Root block: 8211 Filesystem is clean Tree height: 2 Hash function used to sort names: "r5" Objectid map size 2, max 972 Journal parameters: Device [0x0] Magic [0x168ed58c] Size 8193 blocks (including 1 for journal header) (first block 18) Max transaction length 1024 blocks Max batch size 900 blocks Max commit age 30 Blocks reserved by journal: 0 Fs state field: 0x0: sb_version: 2 inode generation number: 0 UUID: 10ae60ee-1abb-49d1-ae55-cf238626c0b5 LABEL: Set flags in SB: ATTRIBUTES CLEAN Super block seems to be correct Something seems seriously wrong here. I'm happy to run any tests or try any patches, this system is mine to play with until the end of the month. Paul Slootman