[Bug 312163] Re: RAID array causing BUG: soft lockup errors/system freeze
I noticed that Pid: x, comm: indexer Not tainted 2.6.27-y-server appeared in all my log files. indexer is part of mnogosearch which I use to search through pdf files. I had built version 3.3.7 on a 2.6.27-9-generic kernel a while ago. I downloaded the new version of mnogosearch, rebuilt it and the system runs fine since. Sorry for misleading -- RAID array causing BUG: soft lockup errors/system freeze https://bugs.launchpad.net/bugs/312163 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 312163] Re: RAID array causing BUG: soft lockup errors/system freeze
@Andy after almost a week of no-crashes, my computer froze last night. Attached is the extract from kernel log. As you asked, I looked for any 'Panic' lines before the crash but there aren't any.. even for the earlier reported crashes. In this crash, the first bug reported is BUG: unable to handle kernel paging request at 0cabf6f5 in process 'apt-config'. After a minute of this occurring, I see several BUG: soft lockup - CPU#0 stuck for 61s! errors. There is also something related to ata1.00: limiting speed to UDMA/44:PIO4 etc., which I don't understand. Is this crash still related to this particular bug report? If not, can you help me identify some other bug against which I can file this? I am sorry but I really don't have the knowledge to look through the filed bugs and correctly identify which one to report this against! I have also installed the kerneloops package and it seems to be submitting all the oops. But I can't figure out where they end up on the kernel oops site. How do I get the URL of my submitted oops!? ** Attachment added: kernel-oops-16-May.txt http://launchpadlibrarian.net/26795599/kernel-oops-16-May.txt -- RAID array causing BUG: soft lockup errors/system freeze https://bugs.launchpad.net/bugs/312163 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 312163] Re: RAID array causing BUG: soft lockup errors/system freeze
We believe that the original bug here is definatly fixed by the change to the lockless pagecache carried in the -14.18 kernel which is now in Intrepid -updates. Closing this one Fix Released for Intrepid. If you do not have the original soft lockup style error then please file a new bug. ** Changed in: linux (Ubuntu Intrepid) Status: Incomplete = Fix Released ** Changed in: linux (Ubuntu Intrepid) Milestone: None = intrepid-updates -- RAID array causing BUG: soft lockup errors/system freeze https://bugs.launchpad.net/bugs/312163 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 312163] Re: RAID array causing BUG: soft lockup errors/system freeze
** Changed in: linux (Ubuntu Intrepid) Importance: Undecided = High -- RAID array causing BUG: soft lockup errors/system freeze https://bugs.launchpad.net/bugs/312163 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 312163] Re: RAID array causing BUG: soft lockup errors/system freeze
@tom -- that appears to be a different symptom. Cirtainly from the one originally reported on this bug, and the one the patch to which I was referring was supposed to fix. Lets get a separate bug filed for that one please. We can dup it to this one should it prove later to be the same. -- RAID array causing BUG: soft lockup errors/system freeze https://bugs.launchpad.net/bugs/312163 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 312163] Re: RAID array causing BUG: soft lockup errors/system freeze
** Also affects: linux (Ubuntu Intrepid) Importance: Undecided Status: New -- RAID array causing BUG: soft lockup errors/system freeze https://bugs.launchpad.net/bugs/312163 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 312163] Re: RAID array causing BUG: soft lockup errors/system freeze
Now that karmic is open the primary task applies there. The fix relating to this issue is already there so closing Fix Released. We continue to monitor Intrepid expecting that the fix is there, in -14 and later. ** Changed in: linux (Ubuntu Intrepid) Status: New = Incomplete ** Changed in: linux (Ubuntu Intrepid) Assignee: (unassigned) = Andy Whitcroft (apw) ** Changed in: linux (Ubuntu) Status: Incomplete = Fix Released -- RAID array causing BUG: soft lockup errors/system freeze https://bugs.launchpad.net/bugs/312163 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 312163] Re: RAID array causing BUG: soft lockup errors/system freeze
As far as I can tell this issue is fixed in Jaunty. I have been really hammering the system for several weeks and have not seen the original problem. I believe some of issues posted here lately are not related to the original bug. The stack traces look different. -- RAID array causing BUG: soft lockup errors/system freeze https://bugs.launchpad.net/bugs/312163 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 312163] Re: RAID array causing BUG: soft lockup errors/system freeze
Is there anything I can do to help solve this problem? -- RAID array causing BUG: soft lockup errors/system freeze https://bugs.launchpad.net/bugs/312163 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 312163] Re: RAID array causing BUG: soft lockup errors/system freeze
@andy: Not so much luck on my side. System was running for less than a day before it crashed again. I've attached a new log file. Looks very similar. ** Attachment added: xfs_oops_2.txt http://launchpadlibrarian.net/26240729/xfs_oops_2.txt -- RAID array causing BUG: soft lockup errors/system freeze https://bugs.launchpad.net/bugs/312163 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 312163] Re: RAID array causing BUG: soft lockup errors/system freeze
I've been having similar problems for quite some time (http://oss.sgi.com/archives/xfs/2009-03/msg00062.html). Lockups affecting random parts of the system. Most of the time, it wouldn't even reboot and I had to pull the plug. I'm running intrepid 2.6.27-11-server x86_64 on an Intel Core2 Duo 2.66GHz with an Adaptec 31605 hardware raid-5 (10x500GB); stripe size 256K. I have attached a log file (xfs_oops.txt) and just upgraded to the -14-server kernel hoping this will solve the problem. ** Attachment added: xfs_oops.txt http://launchpadlibrarian.net/26205627/xfs_oops.txt -- RAID array causing BUG: soft lockup errors/system freeze https://bugs.launchpad.net/bugs/312163 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 312163] Re: RAID array causing BUG: soft lockup errors/system freeze
@Deepak -- sorry missread your version number, jaunty has no -proposed and the main kernels have the fix to this bug. Looking at your log fragment it looks like a different trace. Could you file that one as a new bug. When you do could you check back before the first occurance of the bug in your logs and pick up a say 10 lines before, i am expecting to see a panic or similar before the first one. -- RAID array causing BUG: soft lockup errors/system freeze https://bugs.launchpad.net/bugs/312163 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 312163] Re: RAID array causing BUG: soft lockup errors/system freeze
@tom -- we would expect that to be gone with -14 kernels, if you could report back if your testing is successful that would be helpful. -- RAID array causing BUG: soft lockup errors/system freeze https://bugs.launchpad.net/bugs/312163 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 312163] Re: RAID array causing BUG: soft lockup errors/system freeze
We have deployed 2.6.27-14 on the KVM host. As a caveat we had to add rootdelay=200 to the kernel options as the FC attached storage didn't configure quickly enough for the root fs to be found. This machine is hosting 14 VMs so it's I/O load can be very heavy at times. My potentially related bug list has expanded: bug 268215 bug 289158 bug 300329 bug 303064 bug 312163 bug 348218 -- RAID array causing BUG: soft lockup errors/system freeze https://bugs.launchpad.net/bugs/312163 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 312163] Re: RAID array causing BUG: soft lockup errors/system freeze
It is possible that I am facing this issue too. I am running Jaunty (kernel 2.6.28-11-generic). I have a mdadm based raid too. Here's the dump from /var/log/kern.log that I also posted in: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/367389/comments/2 == Apr 28 14:37:32 cellar kernel: [67396.314920] BUG: unable to handle kernel paging request at 01040424 Apr 28 14:37:32 cellar kernel: [67396.314929] IP: [c02c85cb] prio_tree_insert+0x14b/0x290 Apr 28 14:37:32 cellar kernel: [67396.314940] *pde = Apr 28 14:37:32 cellar kernel: [67396.314945] Oops: [#1] SMP Apr 28 14:37:32 cellar kernel: [67396.314949] last sysfs file: /sys/devices/pci:00/:00:1d.7/usb1/idVendor Apr 28 14:37:32 cellar kernel: [67396.314955] Dumping ftrace buffer: Apr 28 14:37:32 cellar kernel: [67396.314960](ftrace buffer empty) Apr 28 14:37:32 cellar kernel: [67396.314962] Modules linked in: bridge stp bnep vboxnetflt vboxdrv input_polldev video output reiserfs lp snd_intel8x0 snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device iTCO_wdt iTCO_vendor_support snd psmouse soundcore ppdev serio_raw pcspkr usblp snd_page_alloc btusb nvidia(P) intel_agp parport_pc parport agpgart usbhid r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor [last unloaded: uinput] Apr 28 14:37:32 cellar kernel: [67396.315019] Apr 28 14:37:32 cellar kernel: [67396.315024] Pid: 27554, comm: kio_http_cache_ Tainted: P (2.6.28-11-generic #42-Ubuntu) MS-7176 Apr 28 14:37:32 cellar kernel: [67396.315027] EIP: 0060:[c02c85cb] EFLAGS: 00010202 CPU: 0 Apr 28 14:37:32 cellar kernel: [67396.315031] EIP is at prio_tree_insert+0x14b/0x290 Apr 28 14:37:32 cellar kernel: [67396.315034] EAX: ee5b6e3c EBX: 01040404 ECX: f197634c EDX: 010403e0 Apr 28 14:37:32 cellar kernel: [67396.315037] ESI: 0020 EDI: 0001 EBP: f110deb8 ESP: f110de90 Apr 28 14:37:32 cellar kernel: [67396.315039] DS: 007b ES: 007b FS: 00d8 GS: SS: 0068 Apr 28 14:37:32 cellar kernel: [67396.315043] Process kio_http_cache_ (pid: 27554, ti=f110c000 task=f1114b60 task.ti=f110c000) Apr 28 14:37:32 cellar kernel: [67396.315045] Stack: Apr 28 14:37:32 cellar kernel: [67396.315047] f267a3ec f197634c 0116 0118 000433a8 ee5b6e3c f267a3ec Apr 28 14:37:32 cellar kernel: [67396.315056] f267a3c8 ee4e5800 f110dec8 c019bb92 f267a3c8 f1976334 f110ded8 c01a3cc9 Apr 28 14:37:32 cellar kernel: [67396.315065] f267a3c8 f1976334 f110def8 c01a4373 ee8559c0 ee8559b8 f0227aa8 ee8559b8 Apr 28 14:37:32 cellar kernel: [67396.315074] Call Trace: Apr 28 14:37:32 cellar kernel: [67396.315077] [c019bb92] ? vma_prio_tree_insert+0x22/0xc0 Apr 28 14:37:32 cellar kernel: [67396.315084] [c01a3cc9] ? __vma_link_file+0x49/0x80 Apr 28 14:37:32 cellar kernel: [67396.315088] [c01a4373] ? vma_link+0x63/0x90 Apr 28 14:37:32 cellar kernel: [67396.315092] [c01a5a49] ? mmap_region+0x489/0x4e0 Apr 28 14:37:32 cellar kernel: [67396.315097] [c01a5d09] ? do_mmap_pgoff+0x269/0x360 Apr 28 14:37:32 cellar kernel: [67396.315101] [c01084bd] ? sys_mmap2+0xad/0xc0 Apr 28 14:37:32 cellar kernel: [67396.315106] [c0104062] ? syscall_call+0x7/0xb Apr 28 14:37:32 cellar kernel: [67396.315110] [c050] ? relay_hotcpu_callback+0x6d/0xbd Apr 28 14:37:32 cellar kernel: [67396.315117] Code: 8d 48 ff be 01 00 00 00 d3 e6 85 f6 0f 84 17 01 00 00 8b 4d d8 66 85 ff c7 45 ec 00 00 00 00 89 4d f0 74 77 8d 74 26 00 8d 53 dc 8b 42 44 89 45 e8 8b 42 08 2b 42 04 8b 55 e8 c1 e8 0c 8d 7c 02 Apr 28 14:37:32 cellar kernel: [67396.315167] EIP: [c02c85cb] prio_tree_insert+0x14b/0x290 SS:ESP 0068:f110de90 Apr 28 14:37:32 cellar kernel: [67396.315174] ---[ end trace a26021ceb38a2b6e ]--- Apr 28 14:51:09 cellar kernel: [68213.102886] possible SYN flooding on port 41120. Sending cookies. Apr 28 14:53:29 cellar kernel: [68353.332498] BUG: soft lockup - CPU#1 stuck for 61s! [kswapd0:32] Apr 28 14:53:29 cellar kernel: [68353.332502] Modules linked in: bridge stp bnep vboxnetflt vboxdrv input_polldev video output reiserfs lp snd_intel8x0 snd_ac97_codec ac97_bus snd_pcm_oss snd_mixer_oss snd_pcm snd_seq_dummy snd_seq_oss snd_seq_midi snd_rawmidi snd_seq_midi_event snd_seq snd_timer snd_seq_device iTCO_wdt iTCO_vendor_support snd psmouse soundcore ppdev serio_raw pcspkr usblp snd_page_alloc btusb nvidia(P) intel_agp parport_pc parport agpgart usbhid r8169 mii raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear fbcon tileblit font bitblit softcursor [last unloaded: uinput] Apr 28 14:53:29 cellar kernel: [68353.332504] Apr 28 14:53:29 cellar kernel: [68353.332504] Pid: 32, comm: kswapd0 Tainted: P D(2.6.28-11-generic #42-Ubuntu) MS-7176 Apr 28 14:53:29 cellar kernel: [68353.332504] EIP:
[Bug 312163] Re: RAID array causing BUG: soft lockup errors/system freeze
@Deepak -- could you try the kernel in proposed as the -11 kernels do not contain the fix, but the -14 should. Please report back here. -- RAID array causing BUG: soft lockup errors/system freeze https://bugs.launchpad.net/bugs/312163 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 312163] Re: RAID array causing BUG: soft lockup errors/system freeze
@Andy: I added the jaunty-proposed repo (as in https://wiki.ubuntu.com/Testing/EnableProposed) but didn't get any new kernel. I tried both with and without pinning the repos as suggested in 'selective upgrading from -proposed' but still no new kernel. Meanwhile, I am attaching some more kernel logs with lots and lots of oops from early this morning. The system was reporting cpu lockups for almost two hours before I noticed it. When I got to the system, enough applications where misbehaving (error on quit, do not start, can't be killed, etc.) that I had to reboot. Of course, the reboot stuck while leaving X and I then had to power-cycle. ** Attachment added: kernel-oops-29-apr.txt http://launchpadlibrarian.net/26098379/kernel-oops-29-apr.txt -- RAID array causing BUG: soft lockup errors/system freeze https://bugs.launchpad.net/bugs/312163 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 312163] Re: RAID array causing BUG: soft lockup errors/system freeze
I'm testing linux-image-2.6.27-14-server on amd64 KVM this weekend. /etc/apt/sources.list contains: deb http://archive.ubuntu.com/ubuntu/ intrepid-proposed main /etc/apt/preferences contains: Package: * Pin: release a=intrepid-security Pin-Priority: 990 Package: * Pin: release a=intrepid-updates Pin-Priority: 900 Package: * Pin: release a=intrepid-proposed Pin-Priority: 400 Then install the new kernel with: sudo aptitude -t intrepid-proposed install linux-image-2.6.27-14-server linux-image-server and maybe if you have the need for the kernel headers as well: sudo aptitude -t intrepid-proposed install linux- headers-2.6.27-14-server The base header package will be pulled in automatically. Reboot. Compile sys_basher from source code here: http://www.polybus.com/sys_basher_web/ Let two instances of sys_basher -d -ho run over the weekend in GNU screen. One on NFS and the other on xfs. -- RAID array causing BUG: soft lockup errors/system freeze https://bugs.launchpad.net/bugs/312163 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 312163] Re: RAID array causing BUG: soft lockup errors/system freeze
It seems that the upstream fix mentioned above has made it into the Intrepid kernel and is currently in the kernels in Intrepid -proposed. If those who are affected by this could test the kernels from -proposed and report back that would be helpful. Please see https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you in advance! ** Changed in: linux (Ubuntu) Assignee: (unassigned) = Andy Whitcroft (apw) ** Changed in: linux (Ubuntu) Status: Triaged = Incomplete -- RAID array causing BUG: soft lockup errors/system freeze https://bugs.launchpad.net/bugs/312163 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 312163] Re: RAID array causing BUG: soft lockup errors/system freeze
Are the following bugs about the same issue? They all are about soft lockup using xfs on Intrepid: 268215 289158 312163 348218 Thanks. -- RAID array causing BUG: soft lockup errors/system freeze https://bugs.launchpad.net/bugs/312163 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 312163] Re: RAID array causing BUG: soft lockup errors/system freeze
I spent the last few days testing on Jaunty and it seems to be fixed there. Kernel is 2.6.28-11.42 x86_64. I will have more confidence after a few weeks of testing but I think it looks good so far. -- RAID array causing BUG: soft lockup errors/system freeze https://bugs.launchpad.net/bugs/312163 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 312163] Re: RAID array causing BUG: soft lockup errors/system freeze
Chris, it would be great if you could let us know your results if you're either able to test the patch or a newer 2.6.28 based Jaunty kernel. Thanks in advance. -- RAID array causing BUG: soft lockup errors/system freeze https://bugs.launchpad.net/bugs/312163 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 312163] Re: RAID array causing BUG: soft lockup errors/system freeze
** Package changed: mdadm (Ubuntu) = linux (Ubuntu) ** Changed in: linux (Ubuntu) Importance: Undecided = High ** Changed in: linux (Ubuntu) Status: New = Triaged ** Tags added: regression-release -- RAID array causing BUG: soft lockup errors/system freeze https://bugs.launchpad.net/bugs/312163 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 312163] Re: RAID array causing BUG: soft lockup errors/system freeze
I've downgraded to the 2.6.24-16-generic kernel (hardy) and the problem seems to have been resolved. This issue seems to be an issue with the later kernels. -- RAID array causing BUG: soft lockup errors/system freeze https://bugs.launchpad.net/bugs/312163 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 312163] Re: RAID array causing BUG: soft lockup errors/system freeze
The patch that I mentioned above is in the 2.6.28-11 kernel used in Jaunty so it might work also. I'm not sure when I will get a chance to test it though. -- RAID array causing BUG: soft lockup errors/system freeze https://bugs.launchpad.net/bugs/312163 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 312163] Re: RAID array causing BUG: soft lockup errors/system freeze
I've been having the same problem today after adding 2x1TB drives to my Ubuntu x64 system - configured as stripe using mdadm. Random lockups when copying to and reading from raid. Occurs with xfs, reiser and ext3. Will post logs next week. -- RAID array causing BUG: soft lockup errors/system freeze https://bugs.launchpad.net/bugs/312163 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 312163] Re: RAID array causing BUG: soft lockup errors/system freeze
We're still seeing occasional machine check events but nothing logged in /var/log/mcelog Feb 25 00:10:46 dev kernel: [110217.794081] Machine check events logged Feb 25 01:13:16 dev kernel: [113958.998576] Machine check events logged and no further soft lockups. -- RAID array causing BUG: soft lockup errors/system freeze https://bugs.launchpad.net/bugs/312163 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 312163] Re: RAID array causing BUG: soft lockup errors/system freeze
I have been running for over a month now on the older Hardy kernel (2.6.24-23-generic x86_64) and have not seen this issue at all. So it appears it's almost definitely something broken in the newer kernels. At this point it's unclear if it is an Ubuntu or Linux kernel problem. With that said, I found this: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=e8c82c2e23e3527e0c9dc195e432c16784d270fa Has anyone tried this simple patch against the Ubuntu kernel? See also: http://oss.sgi.com/bugzilla/show_bug.cgi?id=805 http://lkml.org/lkml/2009/1/5/307 -- RAID array causing BUG: soft lockup errors/system freeze https://bugs.launchpad.net/bugs/312163 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 312163] Re: RAID array causing BUG: soft lockup errors/system freeze
Hi, I'm seeing a similar problem on my system with frequent soft lockup messages and machine check events logged. Distributor ID: Ubuntu Description:Ubuntu 8.10 Release:8.10 Codename: intrepid Errors with linux-image-2.6.27-11-server (64-bit). It's not clear if there are specific activities triggering the bug. Rolling back to Hardy kernel (linux-image-2.6.24-23-server) seems to have addressed the problem - no error messages in last 12 hours of heavy system activity. System has 2 x Western Digital WDC WD10EADS-00L5B1 drives (1TB) in RAID1 config (for both root partition and swap partition). smulc...@dev:/var/log$ sudo mdadm --misc -D /dev/md1 /dev/md1: Version : 00.90 Creation Time : Mon Feb 9 11:59:43 2009 Raid Level : raid1 Array Size : 974808064 (929.65 GiB 998.20 GB) Used Dev Size : 974808064 (929.65 GiB 998.20 GB) Raid Devices : 2 Total Devices : 2 Preferred Minor : 1 Persistence : Superblock is persistent Update Time : Tue Feb 24 12:18:54 2009 State : active Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 UUID : c8cf85b7:a96d5ddf:b32eea70:c5f0704e Events : 0.25 Number Major Minor RaidDevice State 0 820 active sync /dev/sda2 1 8 181 active sync /dev/sdb2 smulc...@dev:/var/log$ sudo mdadm --misc -D /dev/md2 /dev/md2: Version : 00.90 Creation Time : Mon Feb 9 12:29:38 2009 Raid Level : raid1 Array Size : 1951744 (1906.32 MiB 1998.59 MB) Used Dev Size : 1951744 (1906.32 MiB 1998.59 MB) Raid Devices : 2 Total Devices : 2 Preferred Minor : 2 Persistence : Superblock is persistent Update Time : Tue Feb 24 12:08:02 2009 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 UUID : 8957636b:79f79a80:9069ef6b:43c30843 Events : 0.8 Number Major Minor RaidDevice State 0 810 active sync /dev/sda1 1 8 171 active sync /dev/sdb1 Log file excerpt, Feb 20 10:07:20 dev kernel: [933002.012531] Machine check events logged Feb 20 11:23:39 dev kernel: [937580.632507] BUG: soft lockup - CPU#1 stuck for 61s! [kswapd1:185] Feb 20 11:23:39 dev kernel: [937580.632507] Modules linked in: af_packet ipv6 iptable_filter ip_tables x_tables parport_pc lp parport loop joydev evdev pcspkr container isp1760 amd_rng button i2c_amd756 k8temp shpchp i2c_amd8111 pci_hotplug i2c_core ext3 jbd mbcache sr_mod cdrom pata_acpi sd_mod crc_t10dif sg usbhid hid ata_generic tg3 libphy sata_sil ohci_hcd usbcore pata_amd libata scsi_mod dock raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear md_mod thermal processor fan fbcon tileblit font bitblit softcursor fuse Feb 20 11:23:39 dev kernel: [937580.632507] CPU 1: Feb 20 11:23:39 dev kernel: [937580.632507] Modules linked in: af_packet ipv6 iptable_filter ip_tables x_tables parport_pc lp parport loop joydev evdev pcspkr container isp1760 amd_rng button i2c_amd756 k8temp shpchp i2c_amd8111 pci_hotplug i2c_core ext3 jbd mbcache sr_mod cdrom pata_acpi sd_mod crc_t10dif sg usbhid hid ata_generic tg3 libphy sata_sil ohci_hcd usbcore pata_amd libata scsi_mod dock raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear md_mod thermal processor fan fbcon tileblit font bitblit softcursor fuse Feb 20 11:23:39 dev kernel: [937580.632507] Pid: 185, comm: kswapd1 Tainted: G M 2.6.27-11-server #1 Feb 20 11:23:39 dev kernel: [937580.632507] RIP: 0010:[802abeb7] [802abeb7] find_get_pages+0x77/0x110 Feb 20 11:23:39 dev kernel: [937580.632507] RSP: 0018:88007d5b1bc0 EFLAGS: 0293 Feb 20 11:23:39 dev kernel: [937580.632507] RAX: 880070fd4ae8 RBX: 88007d5b1c00 RCX: 880070fd4ae8 Feb 20 11:23:39 dev kernel: [937580.632507] RDX: RSI: RDI: e261b300 Feb 20 11:23:39 dev kernel: [937580.632507] RBP: 00018031371c R08: R09: 0001 Feb 20 11:23:39 dev kernel: [937580.632507] R10: R11: 0040 R12: 0001 Feb 20 11:23:39 dev kernel: [937580.632507] R13: 802b6ba9 R14: e261b2c0 R15: 0246 Feb 20 11:23:39 dev kernel: [937580.632507] FS: 4020a950() GS:88013f40b300() knlGS:f7bb56b0 Feb 20 11:23:39 dev kernel: [937580.632507] CS: 0010 DS: 0018 ES: 0018 CR0: 8005003b Feb 20 11:23:39 dev kernel: [937580.632507] CR2: 7fa35c00b000 CR3: 00201000 CR4: 06e0 Feb 20 11:23:39 dev kernel: [937580.632507] DR0: DR1: DR2: Feb 20 11:23:39 dev kernel: [937580.632507] DR3: DR6: 0ff0 DR7: 0400 Feb 20 11:23:39 dev kernel: [937580.632507] Feb 20 11:23:39 dev
[Bug 312163] Re: RAID array causing BUG: soft lockup errors/system freeze
I've been seeing the same problem from doing an iozone test on an mdraid 5 array. This keeps appearing in my dmesg: [553927.303743] BUG: soft lockup - CPU#3 stuck for 61s! [iozone:24558] [553927.303755] Modules linked in: wmi video output sbs sbshc pci_slot battery ac bonding iptable_filter ip_tables x_tables parport_pc lp parport loop evdev pcspkr snd_intel8x0 snd_ac97_codec shpchp pci_hotplug k8temp isp1760 ac97_bus snd_pcm snd_timer snd soundcore snd_page_alloc cfi_cmdset_0002 cfi_util container jedec_probe cfi_probe gen_probe button ck804xrom mtd chipreg i2c_nforce2 map_funcs i2c_core ipv6 xfs sr_mod cdrom pata_amd sata_nv sd_mod crc_t10dif sg usb_storage libusual pata_acpi sata_sil24 ata_generic libata scsi_mod ehci_hcd forcedeth ohci_hcd dock usbcore raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear md_mod thermal processor fan fuse vesafb fbcon tileblit font bitblit softcursor [553927.303755] CPU 3: [553927.303755] Modules linked in: wmi video output sbs sbshc pci_slot battery ac bonding iptable_filter ip_tables x_tables parport_pc lp parport loop evdev pcspkr snd_intel8x0 snd_ac97_codec shpchp pci_hotplug k8temp isp1760 ac97_bus snd_pcm snd_timer snd soundcore snd_page_alloc cfi_cmdset_0002 cfi_util container jedec_probe cfi_probe gen_probe button ck804xrom mtd chipreg i2c_nforce2 map_funcs i2c_core ipv6 xfs sr_mod cdrom pata_amd sata_nv sd_mod crc_t10dif sg usb_storage libusual pata_acpi sata_sil24 ata_generic libata scsi_mod ehci_hcd forcedeth ohci_hcd dock usbcore raid10 raid456 async_xor async_memcpy async_tx xor raid1 raid0 multipath linear md_mod thermal processor fan fuse vesafb fbcon tileblit font bitblit softcursor [553927.303755] Pid: 24558, comm: iozone Not tainted 2.6.27-11-server #1 [553927.303755] RIP: 0010:[802abea6] [802abea6] find_get_pages+0x66/0x110 [553927.303755] RSP: 0018:88015ccfd368 EFLAGS: 0246 [553927.303755] RAX: 880048befce0 RBX: 88015ccfd3a8 RCX: 0003 [553927.303755] RDX: 0004 RSI: RDI: e20002055800 [553927.303755] RBP: 88015ccfd318 R08: e2bdafc8 R09: 0008 [553927.303755] R10: 003e R11: 0015dcbf R12: 802b6ba9 [553927.303755] R13: 8801e1450a10 R14: e2bdae40 R15: 0282 [553927.303755] FS: 7f4b4d13c6e0() GS:88025fa2c700() knlGS: [553927.303755] CS: 0010 DS: ES: CR0: 8005003b [553927.303755] CR2: 7fa32cff2e40 CR3: 00015f036000 CR4: 06e0 [553927.303755] DR0: DR1: DR2: [553927.303755] DR3: DR6: 0ff0 DR7: 0400 [553927.303755] [553927.303755] Call Trace: [553927.303755] [802abe83] ? find_get_pages+0x43/0x110 [553927.303755] [802b6a04] ? pagevec_lookup+0x24/0x30 [553927.303755] [a026801d] ? xfs_cluster_write+0xad/0x180 [xfs] [553927.303755] [a0268588] ? xfs_page_state_convert+0x498/0x760 [xfs] [553927.303755] [a02689b1] ? xfs_vm_writepage+0x71/0x120 [xfs] [553927.303755] [802b92f4] ? pageout+0x124/0x270 [553927.303755] [802ab00a] ? page_waitqueue+0xa/0x90 [553927.303755] [802b98ed] ? shrink_page_list+0x34d/0x530 [553927.303755] [802b9c72] ? shrink_inactive_list+0x1a2/0x4b0 [553927.303755] [802b4874] ? get_dirty_limits+0x14/0x2b0 [553927.303755] [802b9ffb] ? shrink_zone+0x7b/0x160 [553927.303755] [802ba16d] ? shrink_zones+0x8d/0x150 [553927.303755] [802ba2b6] ? do_try_to_free_pages+0x86/0x2e0 [553927.303755] [802ba607] ? try_to_free_pages+0x67/0x70 [553927.303755] [802b9120] ? isolate_pages_global+0x0/0x50 [553927.303755] [802b2931] ? __alloc_pages_internal+0x241/0x510 [553927.303755] [802d59ad] ? alloc_pages_current+0xad/0x110 [553927.303755] [802ac417] ? __page_cache_alloc+0x67/0x80 [553927.303755] [802ad053] ? __grab_cache_page+0x63/0xb0 [553927.303755] [80316e49] ? block_write_begin+0x89/0xf0 [553927.303755] [a02674da] ? xfs_vm_write_begin+0x2a/0x30 [xfs] [553927.303755] [a0267050] ? xfs_get_blocks+0x0/0x20 [xfs] [553927.303755] [802ab74c] ? generic_perform_write+0xbc/0x1c0 [553927.303755] [802ad4b2] ? generic_file_buffered_write+0x92/0x170 [553927.303755] [a02702e3] ? xfs_write+0x6b3/0x9b0 [xfs] [553927.303755] [a026fc31] ? xfs_write+0x1/0x9b0 [xfs] [553927.303755] [a026bc98] ? xfs_file_aio_write+0x58/0x60 [xfs] [553927.303755] [802e98a9] ? do_sync_write+0xf9/0x140 [553927.303755] [803a7cd1] ? __rb_erase_color+0x101/0x1d0 [553927.303755] [80266fd0] ? autoremove_wake_function+0x0/0x40 [553927.303755] [80387011] ? aa_file_permission+0x21/0xf0 [553927.303755] [80387138] ? apparmor_file_permission+0x28/0x30 [553927.303755]
[Bug 312163] Re: RAID array causing BUG: soft lockup errors/system freeze
I'm still testing but I have been using the kernel from 8.04 Hardy 2.6.24-23-generic and problem has not appeared for several days now. I'm still running Intrepid but with the older kernel. If this is true then this bug must have been introduced somewhere between 2.6.24 and 2.6.27 (and as I mentioned 2.6.28 is affected as well). -- RAID array causing BUG: soft lockup errors/system freeze https://bugs.launchpad.net/bugs/312163 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 312163] Re: RAID array causing BUG: soft lockup errors/system freeze
I built the current Jaunty kernel (2.6.28-4-generic) and tried that but it has the same problem. It seems the error happens much more often when using XFS but I'm not sure if it's related to XFS or not. I have found that doing very long running large SqLite operations is the best way to recreate the error (disk I/O is very high). Sometimes it can take hours of hammering the disks before the error appears (race condition I'm guessing). Often it hard-locks the OS and the RAID arrays have to be resynced (which takes several hours) after reset/reboot. -- RAID array causing BUG: soft lockup errors/system freeze https://bugs.launchpad.net/bugs/312163 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 312163] Re: RAID array causing BUG: soft lockup errors/system freeze
I've been seeing this same problem. I have two arrays one raid0 other raid5 both with xfs on them. If I copy 10GB files between the two theres a 50% chance it'll pause and end with cpu soft lockup messages. Unsure if this is related: http://www.nabble.com/Re:-BUG:-soft-lockup---is-this-XFS-problem--td21148012.html -- RAID array causing BUG: soft lockup errors/system freeze https://bugs.launchpad.net/bugs/312163 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 312163] Re: RAID array causing BUG: soft lockup errors/system freeze
After more testing I have modified the original description because originally I was thinking it just affected XFS but now I don't think that is the case. I just got another crash when using Ext3 so now I'm not sure where the problem might be. It still looks like it probably has something to do with the filesystem layers though. In this latest case all I did was start a 'grep' on a file and the CPU went to 100% and the process froze completely (could not be killed even with kill -KILL). A few seconds later I started getting the BUG: soft lockup errors. I have attached the latest error. ** Summary changed: - XFS on RAID array causing BUG: soft lockup errors/system freeze + RAID array causing BUG: soft lockup errors/system freeze ** Description changed: Running Ubuntu 8.10 Intrepid Desktop 64-bit with all current updates. Kernel is 2.6.27-9-generic #1 SMP Thu Nov 20 22:15:32 UTC 2008 x86_64 GNU/Linux. System is Abit IP35-Pro motherboard, Q9550 CPU, 8 GB RAM, 4 Western Digital WD6401AALS-00L3B2 drives (640 GB, SATA). - MemTest and drive tests returns no errors. Using a RAID5 array across the 4 drives with XFS as the file system. + MemTest and drive tests returns no errors. Using a RAID5 array across the 4 drives with XFS and Ext3 as the file systems. It seems when the system gets under heavy I/O load on the RAID array it will eventually freeze (requiring a hard reboot). The errors are inconsistent in how long they take to show up and the specific applications causing the I/O load don't seem to matter but the error always shows up eventually. It has already happened dozens of times after I start a load on the machine. Examples of scenarios where the bug has appeared: * Running Bonnie++ benchmarks. I have seen this kill the system within 30 seconds. Other times it causes no problems. * Running a large SQlite import while simultaneously tar/gzip'ing a large directory structure. * Running a large MySql import while tar/gzip'ing a large directory structure. * Multiple tar/gzip processes running at the same time. The specific process that gets the CPU: soft lockup error varies. I have seen it lockup in kswapd, pdflush, gzip, sqlite3, bonnie++ (see attached logs). - The error appears to be related to XFS because all the stack traces have - XFS calls in them. I have attached some of my stack traces. They - repeat many times and I have only included the first distinct events - before my system crashed. + I have attached some of my stack traces. They repeat many times and I + have only included the first distinct events before my system crashed. ** Attachment added: lockups2.txt http://launchpadlibrarian.net/20864107/lockups2.txt -- RAID array causing BUG: soft lockup errors/system freeze https://bugs.launchpad.net/bugs/312163 You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
