Well, the crash happens still, up to twice a week. It is still somehow related to heavy disk use, as it always happens when I'm copying multiple large files to RAIDSET-1, or during a nightly backup, which copies lots of files from RAIDSET-1 to RAIDSET-0.
This morning the ext3-journal crashed again, and I found an extra interesting log-entry. See below. I'm referring to the IO error a remount reports. Clues anyone? Current kernel: einstein:~# uname -a Linux einstein 2.6.11-1-686-smp #1 SMP Mon Jun 20 20:18:45 MDT 2005 i686 GNU/Linux einstein:~# dpkg -l | grep kernel-image-2.6.11 ii kernel-image-2.6.11-1-686-smp 2.6.11-7 Linux kernel image for version 2.6.11 on PPr The log below was captured during nightly backup, copying a large number of files from RAIDSET-1 to RAIDSET-0. From kern.log: Aug 23 04:18:29 einstein kernel: EXT3-fs error (device dm-2): ext3_readdir: bad entry in directory #68272136: rec_len %% 4 != 0 - offset=0, inode=410831438 Aug 23 04:18:29 einstein kernel: Aborting journal on device dm-2. Aug 23 04:18:30 einstein kernel: ext3_abort called. Aug 23 04:18:30 einstein kernel: EXT3-fs error (device dm-2): ext3_journal_start_sb: Detected aborted journal Aug 23 04:18:30 einstein kernel: Remounting filesystem read-only Aug 23 04:18:40 einstein kernel: EXT3-fs error (device dm-1): ext3_free_blocks: Freeing blocks not in datazone - block = 4124995008, count = 1 Aug 23 04:18:40 einstein kernel: Aborting journal on device dm-1. Aug 23 04:18:40 einstein kernel: EXT3-fs error (device dm-1): ext3_free_blocks: Freeing blocks not in datazone - block = 3676589613, count = 1 Aug 23 04:18:40 einstein kernel: EXT3-fs error (device dm-1): ext3_free_blocks: Freeing blocks not in datazone - block = 2129345344, count = 1 Aug 23 04:18:40 einstein kernel: EXT3-fs error (device dm-1): ext3_free_blocks: Freeing blocks not in datazone - block = 2512093470, count = 1 Aug 23 04:18:40 einstein kernel: EXT3-fs error (device dm-1): ext3_free_blocks: Freeing blocks not in datazone - block = 134217728, count = 1 Aug 23 04:18:40 einstein kernel: EXT3-fs error (device dm-1) in ext3_reserve_inode_write: Journal has aborted Aug 23 04:18:40 einstein kernel: EXT3-fs error (device dm-1) in ext3_truncate: Journal has aborted Aug 23 04:18:40 einstein kernel: EXT3-fs error (device dm-1) in ext3_reserve_inode_write: Journal has aborted Aug 23 04:18:40 einstein kernel: EXT3-fs error (device dm-1) in ext3_orphan_del: Journal has aborted Aug 23 04:18:40 einstein kernel: EXT3-fs error (device dm-1) in ext3_reserve_inode_write: Journal has aborted Aug 23 04:18:40 einstein kernel: EXT3-fs error (device dm-1) in ext3_delete_inode: Journal has aborted Aug 23 04:18:40 einstein kernel: __journal_remove_journal_head: freeing b_committed_data Aug 23 04:18:40 einstein last message repeated 2 times Aug 23 04:18:40 einstein kernel: ext3_abort called. Aug 23 04:18:40 einstein kernel: EXT3-fs error (device dm-1): ext3_journal_start_sb: Detected aborted journal Aug 23 04:18:40 einstein kernel: Remounting filesystem read-only ...REBOOT and REMOUNT... Aug 23 05:42:30 einstein kernel: EXT3-fs warning (device dm-1): ext3_clear_journal_err: Filesystem error recorded from previous mount: IO failure Aug 23 05:42:30 einstein kernel: EXT3-fs warning (device dm-1): ext3_clear_journal_err: Marking fs in need of filesystem check. Aug 23 05:42:30 einstein kernel: EXT3-fs warning: mounting fs with errors, running e2fsck is recommended Aug 23 05:42:30 einstein kernel: EXT3 FS on dm-1, internal journal Aug 23 05:42:30 einstein kernel: EXT3-fs: recovery complete. Aug 23 05:42:30 einstein kernel: EXT3-fs: mounted filesystem with ordered data mode. Aug 23 05:43:17 einstein kernel: kjournald starting. Commit interval 5 seconds Aug 23 05:43:17 einstein kernel: EXT3-fs warning (device dm-2): ext3_clear_journal_err: Filesystem error recorded from previous mount: IO failure Aug 23 05:43:17 einstein kernel: EXT3-fs warning (device dm-2): ext3_clear_journal_err: Marking fs in need of filesystem check. Aug 23 05:43:17 einstein kernel: EXT3-fs warning: mounting fs with errors, running e2fsck is recommended Aug 23 05:43:17 einstein kernel: EXT3 FS on dm-2, internal journal Aug 23 05:43:17 einstein kernel: EXT3-fs: recovery complete. Aug 23 05:43:17 einstein kernel: EXT3-fs: mounted filesystem with ordered data mode. Regards, Jeroen
