Re: 3Ware 9000 series hangs under load
Jo Rhett wrote: > The driver logs all useful stuff, and the SEC logfile surfer does a > good job of notifying you quickly. I can send you an SEC > configuration for that if you want. Hm - what is SEC? -- Oliver Lehmann http://www.pofo.de/ http://wishlist.ans-netz.de/ ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: fifo log problem
At 02:16 AM 11/10/2008, Poul-Henning Kamp wrote: In message <[EMAIL PROTECTED]>, Mike Tancsa writes: >I tried changing the config so that there is only the fifo log being >written to and disabled newsyslog so that syslogd is not getting a >HUP signal. The strange thing is that reading from it gives >different results?!? > >Sometimes doing >[ps0278]# fifolog_reader all.fifo | wc >>From0 Wed Dec 31 19:00:00 1969 >To 1225760679 Mon Nov 3 20:04:39 2008 >Read from 1d800 > 59 4133068 >0[ps0278]# > >and a exactly for 1min it will show the correct results > >0[ps0278]# fifolog_reader all.fifo | wc >>From0 Wed Dec 31 19:00:00 1969 >To 1225760538 Mon Nov 3 20:02:18 2008 >Read from 0 >10765 75995 556816 >0[ps0278]# I could fear that you have two fifologs running at the same time, possibly as a result of syslogd doing something strange on sighup... There is nothing really strange about the config. Any idea on how to resolve? ---Mike ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: problem moving gmirror between two machines.
On Sun, 16 Nov 2008 21:29 -, hartzell wrote: George Hartzell writes: > > I have an HP DL360 with a pair of 1TB seagate disks that's been > running -STABLE with a ZFS root partition set up using the tools > available here: > > http://yds.coolrat.org/zfsboot.shtml > > It's been working great. As part of trying to understand what's going > on, I csup'ed to -RELENG earlier today and rebuilt/installed the > kernel and world whilst running on the DL360, so everything should be > current. > > I tried to move the disks into an HP DL320 G4 and it fails to boot > because it can't find /dev/mirror/boot (which it wants to mount onto > /strap and then parts get nullfs'ed onto /boot and /rescue). It gives > me the opportunity to start a shell, and from that shell I can do a > zfs mount -a and get all of the zfs filesystems mounted, but there's > nothing in /dev/mirror. No gmirror status and list are silent. > > I can move the disks back into the older machine and they work fine. > > I've run fdisk -s ad4 and bsdlabel -A /dev/ad4s1a and diffed the > output from the two machines and they're identical. > > I've booted with kern.geom.mirror.debug=2 and the DL320G4 tastes > /dev/ad4s1a (along with everything else) but doesn't do anything with > it. > > Any ideas? > [for the archives] Solved. gmirror had been set up with -h specifying the device, and although the newer server used the same device names for its disks (ad[46]) it assigned them to different hot swap bays. Once I switched the disks everything came up fine. g. Wouldn't it be more feasible in this situation to just glabel the disks and mount them from /dev// instead. Might make your life easier in the future for swapping disks. Example: /dev/ufs/tmp on /tmp (ufs, asynchronous, local, gjournal) /dev/ufs/usr on /usr (ufs, local, soft-updates) /dev/ufs/var on /var (ufs, local, soft-updates) -- Sincerely,-- Jason H. ;; Networked Systems Engineering. The Command Line Kid.;; Multi-user Systems Advocate. mailto:gmail.com!cmdlnkid ;; 1(616)403- / BSD Group. - (2^(N-1)) ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: problem moving gmirror between two machines.
George Hartzell writes: > > I have an HP DL360 with a pair of 1TB seagate disks that's been > running -STABLE with a ZFS root partition set up using the tools > available here: > > http://yds.coolrat.org/zfsboot.shtml > > It's been working great. As part of trying to understand what's going > on, I csup'ed to -RELENG earlier today and rebuilt/installed the > kernel and world whilst running on the DL360, so everything should be > current. > > I tried to move the disks into an HP DL320 G4 and it fails to boot > because it can't find /dev/mirror/boot (which it wants to mount onto > /strap and then parts get nullfs'ed onto /boot and /rescue). It gives > me the opportunity to start a shell, and from that shell I can do a > zfs mount -a and get all of the zfs filesystems mounted, but there's > nothing in /dev/mirror. No gmirror status and list are silent. > > I can move the disks back into the older machine and they work fine. > > I've run fdisk -s ad4 and bsdlabel -A /dev/ad4s1a and diffed the > output from the two machines and they're identical. > > I've booted with kern.geom.mirror.debug=2 and the DL320G4 tastes > /dev/ad4s1a (along with everything else) but doesn't do anything with > it. > > Any ideas? > [for the archives] Solved. gmirror had been set up with -h specifying the device, and although the newer server used the same device names for its disks (ad[46]) it assigned them to different hot swap bays. Once I switched the disks everything came up fine. g. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: FreeBSD 7-STABLE, isp(4), QLE2462: panic & deadlocks
On Wed, Oct 15, 2008 at 08:54:53PM +0300, Panagiotis Christias wrote: > On Wed, Oct 15, 2008 at 09:44:15AM +0400, Oleg Sharoiko wrote: > > Hi! > > > > On Wed, 2008-10-15 at 01:23 +0300, Panagiotis Christias wrote: > > > > > However, when we connect them to the CX3-40, create and mount a new > > > partition and then do something as simple as "tar -C /san -xf ports.tgz" > > > the system panics and deadlocks. We have tried several FreeBSD versions > > > (6.3 i386/adm64, 7.0 i386/adm64, 7.1 i386/adm64 and lastly 7-STABLE i386 > > > - we also tried the latest 8-CURRENT snapshot but it panicked too soon). > > > The result is always the same; panic and deadlock. > > > > Try reducing the number of "tagged openings" with 'camcontrol tags' down > > to 46. If it doesn't work try reducing it further to 2. Also be advised > > that I've seen panics with geom_multipath in FreeBSD-7, unfortunately I > > had no time to test it in -current. > > > Hm.. that would probably explain the fact that I was unable to panic the > system when I had set the hint.isp.0.debug="0x1F" in /boot/device.hints. > > Currently I am stress testing the server with the tagged openings set to > 44 (first value tested). Until now there is no panic or deadlock. I am > trying concurrent tar extractions and rsync copies. The filesystem looks > ok till now according to fsck. I will let it write/copy/delete overnight > and tomorrow I will try different tagged opening values. > > Thank you for the hint! I am wondering what is the performance penalty > with decreased tagged openings. Also, is there anything else I could try > in order to get more useful debug output? I have at least three servers > that I could use for any kind of tests and I am willing to spend as much > time I can get to help solving the problem. > > Finally, the only output in the logs is: > > Expensive timeout(9) function: 0xc06f4210(0xc67e1200) 0.059422635 s > Expensive timeout(9) function: 0xc08d4fd0(0) 0.060676147 s > > I suppose that is related to the CAMDEBUG kernel config options. For the record, I have done many tests using several stressing tools in parallel, different FreeBSD versions (up to 7.1beta2), various filesystem configurations (plain ufs2 with softupdates, ufs2 and gjournal, zfs) and various tag openings values (down to 2). Regardless of the configuration, the system deadlocks, panics or the filesystem gets awfully corrupted within seconds, minutes or a few hours. The only configuration that seems to work without problems(?) but with a unacceptable *severe* performance penalty is when tag openings are set to minimum value of 2 (that is more or less same as disabling tagged command queueing at all). All tests ran using a 500 GB RAID5 LUN on an EMC Clariion CX340: da0 at isp0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-4 device da0: Serial Number CK200083100148 da0: 400.000MB/s transfers da0: Command Queueing Enabled da0: 512000MB (1048576000 512 byte sectors: 255H 63S/T 65270C) Previously, a Sun StorEdge T3 was tested which worked flawlessly but it had a 1 Gbps fibre channel interface, instead of a 4 Gbps that Clariion has, was recognized as a SCSI-3 device and had 2 tags openings (no surprise) by default: da1 at isp1 bus 0 target 0 lun 0 da1: Fixed Direct Access SCSI-3 device da1: 100.000MB/s transfers da1: 241724MB (495050752 512 byte sectors: 255H 63S/T 30815C) As I mentioned before, I am willing to spend time or/and provide access to the system for testing and debugging. Regards, Panagiotis -- Panagiotis J. ChristiasNetwork Management Center [EMAIL PROTECTED]National Technical Univ. of Athens, GREECE ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: 3Ware 9000 series hangs under load
On Nov 12, 2008, at 12:37 PM, Philip Murray wrote: I just installed sysutils/tw_cli from ports, and it sets up some 'periodic' scripts for you. To be precise it puts 407.status-3ware- raid in /usr/local/etc/periodic/daily Don't use that. It's a very old version of the code. Use the binary version of tw_cli that matches the firmware on your controller. -- Jo Rhett Net Consonance : consonant endings by net philanthropy, open source and other randomness ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: 3Ware 9000 series hangs under load
Philip Murray wrote: Anyway, I stopped running 3dmd (or 3dm2 I think it's called now) to monitor it, and the crashes went away. It's had hundreds of days uptime since. We have never used 3dm2, and the 9500 units have been rock solid for us. I've never been game enough to try newer versions of 3dm, but a cronjob of tw_cli allows me to monitor it now without the lockups. Might not be your problem, but it's worth a shot if all else fails. The driver logs all useful stuff, and the SEC logfile surfer does a good job of notifying you quickly. I can send you an SEC configuration for that if you want. -- Jo Rhett Net Consonance : consonant endings by net philanthropy, open source and other randomness ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: 6.4-RC2 crashes after a few minutes of uptime
On 2008-11-16, at 17:28 , barbara wrote: Hi Rory, did you see my replies or are you missing them for any reason? Yes, I have seen your replies. I must have missed the PR you mentioned last time, sorry. No problem! Your panics and some aspects about how they happens look like mine to me, look here: http: //lists.freebsd.org/pipermail/freebsd-stable/2008-October/ 045865.html Yes, indeed. That looks very similar to the issue I'm running into with 6.4-RC2 as well. Sounds like it might be a regression in ata(4). At least you were able to open the core dump. Are you still able to open core dumps with RC2? I'm not sure. I'm running STABLE and I had no panics after the branch has changed to RC2. It seems that my panics are not frequent as yours. Anyway my box freezed a couple of times after last newvers.sh and the symptoms looked like the same, with messages about acd0. I was able to ping it but it won't let me ssh in, like it was using all the cpus. About kgdb... I never used freebsd-update, so sorry if I'm saying something stupid, but could it be the case that the kernel has been built without debugging symbols or something like that? Does freebsd- update provide a kernel.debug? I haven't had to use a the kernel.debug file in the obj dir in a long time. As far as I know, these days, the GENERIC kernel includes debug symbols. And in cases when there aren't any debug symbols, that shouldn't prevent kgdb from loading, I wouldn't think. - rory ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: qlogic qle2462 hba and freebsd stable on a dl360 g5
On Thu, Nov 13, 2008 at 12:22:11PM +0100, Claus Guttesen wrote: > Hi. > > I'm looking at a qlogic qle2462 hba for my dl360 g5. The thread > http://www.mail-archive.com/freebsd-stable@freebsd.org/msg99497.html > mentions a deadlock when system is loaded. Has this issue been > resolved? Are there other PCI Express hba's which are known to work > with freebsd stable and dl360 g5? Hello, no, the issue has not been resolved. The system still deadlocks regardless the value of tag openings (even when set to the minimum value of 2) and the filesystem gets corrupted or totally destroyed. I am still looking for a solution and willing to do any tests. Regards, Panagiotis -- Panagiotis J. ChristiasNetwork Management Center [EMAIL PROTECTED]National Technical Univ. of Athens, GREECE ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: 6.4-RC2 crashes after a few minutes of uptime
> > Hi Rory, > > > > did you see my replies or are you missing them for any reason? > > Yes, I have seen your replies. I must have missed the PR you mentioned > last time, sorry. No problem! > > Your panics and > > some aspects about how they happens look like mine to me, look here: > > http: > > //lists.freebsd.org/pipermail/freebsd-stable/2008-October/045865.html > > Yes, indeed. That looks very similar to the issue I'm running into > with 6.4-RC2 as well. Sounds like it might be a regression in ata(4). > At least you were able to open the core dump. Are you still able to > open core dumps with RC2? > I'm not sure. I'm running STABLE and I had no panics after the branch has changed to RC2. It seems that my panics are not frequent as yours. Anyway my box freezed a couple of times after last newvers.sh and the symptoms looked like the same, with messages about acd0. I was able to ping it but it won't let me ssh in, like it was using all the cpus. About kgdb... I never used freebsd-update, so sorry if I'm saying something stupid, but could it be the case that the kernel has been built without debugging symbols or something like that? Does freebsd-update provide a kernel.debug? I've seen that you are not using shiny quad-core, but could you try building a kernel by yourself? I think that you could do it using a different, more powerful, freebsd box if you have it, or even on qemu. I could help if you wish. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: 6.4-RC2 crashes after a few minutes of uptime
On 2008-11-16, at 7:24 , Barbara wrote: Ok, guess something is amiss with the CD-ROM drive on this notebook, as in GNOME, it flashes an icon of a CD on the desktop from time to time, as if it has detected a disc in the drive. But of course there is no disc in the drive. I believe it did the same with 6.3 though, but as said before didn't ever panic due to this issue. So, some anecdotal info, after running RC2 for a few days now. It seems the pattern is that it seems to always panic a few minutes after a first cold boot, but then seems to remain stable after the second boot. Odd, as with 6.3 this didn't happen. So, I happened to catch a panic while working in the syscons console after one of these cold boots. As far as I can tell, the panic does have something to do with the the CD-ROM drive, as right after I saw this message on the console, it immediately paniced: acd0: WARNING - PREVENT_ALLOW read data overrun 18>0 and then the panic is as follows: kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode fault virtual address = 0x78 fault code = supervisor read, page not present instruction pointer = 0x20:0xc06d39b9 stack pointer = 0x28:0xca865c10 frame pointer = 0x28: 0xca865c14 code segment= base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags= resume, IOPL = 0 current process = 19 (swi6: task queue) trap number = 12 panic: page fault Uptime: 1h9m7s Physical memory: 179MB Dumping 43MB: 28 12 Dump complete Hi Rory, did you see my replies or are you missing them for any reason? Yes, I have seen your replies. I must have missed the PR you mentioned last time, sorry. Your panics and some aspects about how they happens look like mine to me, look here: http: //lists.freebsd.org/pipermail/freebsd-stable/2008-October/045865.html Yes, indeed. That looks very similar to the issue I'm running into with 6.4-RC2 as well. Sounds like it might be a regression in ata(4). At least you were able to open the core dump. Are you still able to open core dumps with RC2? Unfortunately I got no answer about that and I've had no comment in the pr I've filed http://www.freebsd.org/cgi/query-pr.cgi?pr=128076 I wonder if someone had the time to look at it. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
6.4-RC2 crashes after a few minutes of uptime
>Ok, guess something is amiss with the CD-ROM drive on this notebook, >as in GNOME, it flashes an icon of a CD on the desktop from time to >time, as if it has detected a disc in the drive. But of course there >is no disc in the drive. I believe it did the same with 6.3 though, >but as said before didn't ever panic due to this issue. > >So, some anecdotal info, after running RC2 for a few days now. It >seems the pattern is that it seems to always panic a few minutes after >a first cold boot, but then seems to remain stable after the second >boot. Odd, as with 6.3 this didn't happen. So, I happened to catch a >panic while working in the syscons console after one of these cold >boots. As far as I can tell, the panic does have something to do with >the the CD-ROM drive, as right after I saw this message on the >console, it immediately paniced: > >acd0: WARNING - PREVENT_ALLOW read data overrun 18>0 > >and then the panic is as follows: > >kernel trap 12 with interrupts disabled > >Fatal trap 12: page fault while in kernel mode >fault virtual address = 0x78 >fault code = supervisor read, page not present >instruction pointer= 0x20:0xc06d39b9 >stack pointer = 0x28:0xca865c10 >frame pointer = 0x28: 0xca865c14 >code segment = base 0x0, limit 0xf, type 0x1b > = DPL 0, pres 1, def32 1, gran 1 >processor eflags = resume, IOPL = 0 >current process = 19 (swi6: task queue) >trap number= 12 >panic: page fault >Uptime: 1h9m7s >Physical memory: 179MB >Dumping 43MB: 28 12 >Dump complete Hi Rory, did you see my replies or are you missing them for any reason? Your panics and some aspects about how they happens look like mine to me, look here: http: //lists.freebsd.org/pipermail/freebsd-stable/2008-October/045865.html Unfortunately I got no answer about that and I've had no comment in the pr I've filed http://www.freebsd.org/cgi/query-pr.cgi?pr=128076 I wonder if someone had the time to look at it. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: hangs for 7.1-PRE [was: problem possibly related to multi-ip jail patch?]
On Sun, 16 Nov 2008, Lorenzo Perone wrote: Hi, I've been experiencing problems with one of the machines running FreeBSD 7.1-PRERELEASE #2: Thu Oct 16 20:23:09 CEST 2008 with the multi-ip patch bz_jail7-20080920-01-at150161.diff, and I'm wondering if it possibly related to the patch - in any case, any advice would be very welcome. bottom line is that most of this looks less likely to be a jail problem. It happens that mysql (tried both 4.0 and 5.1, in 2 separate jails), at some time stop responding to connections, and mysql gets stuck in sbwait state. It is only killable with kill -9 Yeah, I had been seeing mysql hang or go to 99% CPU for years once in a while; it's been more rare the last months. I have seen it in- and outside of jails, with or without patches. You could try to see if you can get backtraces of those processes. each of the two mysqlds is running in a jail on one private IP, serving connections to a webserver nearby - the latter having one public and one private IP, communicating with the other jail via the private network. I also experienced two complete system hangs (which must not be necessarily related to the mysql problem) both during a shutdown -r now. one was a panic, in another case the machine was still pingable but did not shut down completely. I could only reset it over the DRAC. here's a screenshot I made over the Dell RAC: http://lorenzo.yellowspace.net/stuck.png Looking at your image I see more problems before the shutdown so this as well is most likely not a jail problem. Since I'm also using zfs there and the kernel has been built with the DTRACE options. any advice (also about which more details that I should/could provide) would be very welcome... I am Cc:ing the answer to stable@ and setting reply-to: to move the discussion there. /bz -- Bjoern A. Zeeb Stop bit received. Insert coin for new game. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: 6.4-RC2 crashes after a few minutes of uptime
On 2008-11-07, at 12:17 , Ken Smith wrote: On Fri, 2008-11-07 at 00:00 -0500, Rory Arms wrote: Well, if I can assist with further debugging, let me know. The person who followed up with a list of things that *may* have made the problem go away mentioned one of the things was disabling powerd. Do you have that enable, and if yes would you mind disabling it to see if that's the culprit? Ken, Ok, guess something is amiss with the CD-ROM drive on this notebook, as in GNOME, it flashes an icon of a CD on the desktop from time to time, as if it has detected a disc in the drive. But of course there is no disc in the drive. I believe it did the same with 6.3 though, but as said before didn't ever panic due to this issue. So, some anecdotal info, after running RC2 for a few days now. It seems the pattern is that it seems to always panic a few minutes after a first cold boot, but then seems to remain stable after the second boot. Odd, as with 6.3 this didn't happen. So, I happened to catch a panic while working in the syscons console after one of these cold boots. As far as I can tell, the panic does have something to do with the the CD-ROM drive, as right after I saw this message on the console, it immediately paniced: acd0: WARNING - PREVENT_ALLOW read data overrun 18>0 and then the panic is as follows: kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode fault virtual address = 0x78 fault code = supervisor read, page not present instruction pointer = 0x20:0xc06d39b9 stack pointer = 0x28:0xca865c10 frame pointer = 0x28:0xca865c14 code segment= base 0x0, limit 0xf, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags= resume, IOPL = 0 current process = 19 (swi6: task queue) trap number = 12 panic: page fault Uptime: 1h9m7s Physical memory: 179MB Dumping 43MB: 28 12 Dump complete This is also the computer, as you may recall, that I can't ever get kgdb(1) to open the core dump file. Note the uptime on that particular boot was 1h because I pretty much let it sit idle after booting. So, gdm loaded and then I switched to syscons, logged in, and then pretty much let it idle, till it paniced. Hope that helps, - rory ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
