Re: Crashes on X7SPE-HF with em

[Omer Faruk SEN]

Maybe it is related with
http://unix.derkeiler.com/Mailing-Lists/FreeBSD/net/2010-01/msg00021.html
which is still not in 8-STABLE

On Fri, Sep 17, 2010 at 12:36 PM, Philipp Wuensche  wrote:
> In the end this turned out to be faulty hardware, at least the mainboard
> died a tragic death and has to be replaced now.
>
> Thanks for the help anyway and sorry for the noise!
>
> Greetings,
> philipp
>
> Philipp Wuensche wrote:
>> Hi,
>>
>> I'm having trouble with a system on a Supermicro X7SPE-HF, it crashes
>> about once a day. I haven't found a way to trigger this yet.
>>
>> The system has a bunch of VLANs on em1, it does routing between them.
>>
>> Currently its running 8-STABLE but it happend with 8.1-RELEASE too.
>>
>> greetings,
>> Philipp
>>
>> # kgdb kernel.debug /var/crash/vmcore.0
>> GNU gdb 6.1.1 [FreeBSD]
>> Copyright 2004 Free Software Foundation, Inc.
>> GDB is free software, covered by the GNU General Public License, and you are
>> welcome to change it and/or distribute copies of it under certain
>> conditions.
>> Type "show copying" to see the conditions.
>> There is absolutely no warranty for GDB.  Type "show warranty" for details.
>> This GDB was configured as "amd64-marcel-freebsd"...
>>
>> Unread portion of the kernel message buffer:
>>
>>
>> Fatal trap 9: general protection fault while in kernel mode
>> cpuid = 0; apic id = 00
>> instruction pointer   = 0x20:0x8061f5a8
>> stack pointer         = 0x28:0xff8e64d0
>> frame pointer         = 0x28:0xff8e64e0
>> code segment          = base 0x0, limit 0xf, type 0x1b
>>                       = DPL 0, pres 1, long 1, def32 0, gran 1
>> processor eflags      = interrupt enabled, resume, IOPL = 0
>> current process               = 0 (em1 taskq)
>> trap number           = 9
>> panic: general protection fault
>> cpuid = 0
>> Uptime: 13h24m39s
>> Physical memory: 4079 MB
>> Dumping 1349 MB: 1334 1318 1302 1286 1270 1254 1238 1222 1206 1190 1174
>> 1158 1142 1126 1110 1094 1078 1062 1046 1030 1014 998 982 966 950 934
>> 918 902 886 870 854 838 822 806 790 774 758 742 726 710 694 678 662 646
>> 630 614 598 582 566 550 534 518 502 486 470 454 438 422 406 390 374 358
>> 342 326 310 294 278 262 246 230 214 198 182 166 150 134 118 102 86 70 54
>> 38 22 6
>>
>> Reading symbols from /boot/kernel/zfs.ko...Reading symbols from
>> /boot/kernel/zfs.ko.symbols...done.
>> done.
>> Loaded symbols for /boot/kernel/zfs.ko
>> Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from
>> /boot/kernel/opensolaris.ko.symbols...done.
>> done.
>> Loaded symbols for /boot/kernel/opensolaris.ko
>> Reading symbols from /boot/kernel/coretemp.ko...Reading symbols from
>> /boot/kernel/coretemp.ko.symbols...done.
>> done.
>> Loaded symbols for /boot/kernel/coretemp.ko
>> Reading symbols from /boot/kernel/ahci.ko...Reading symbols from
>> /boot/kernel/ahci.ko.symbols...done.
>> done.
>> Loaded symbols for /boot/kernel/ahci.ko
>> Reading symbols from /boot/kernel/ipmi.ko...Reading symbols from
>> /boot/kernel/ipmi.ko.symbols...done.
>> done.
>> Loaded symbols for /boot/kernel/ipmi.ko
>> Reading symbols from /boot/kernel/smbus.ko...Reading symbols from
>> /boot/kernel/smbus.ko.symbols...done.
>> done.
>> Loaded symbols for /boot/kernel/smbus.ko
>> Reading symbols from /boot/kernel/pflog.ko...Reading symbols from
>> /boot/kernel/pflog.ko.symbols...done.
>> done.
>> Loaded symbols for /boot/kernel/pflog.ko
>> Reading symbols from /boot/kernel/pf.ko...Reading symbols from
>> /boot/kernel/pf.ko.symbols...done.
>> done.
>> Loaded symbols for /boot/kernel/pf.ko
>> #0  doadump () at pcpu.h:224
>> 224           __asm("movq %%gs:0,%0" : "=r" (td));
>> (kgdb) list *0x8061f5a8
>> 0x8061f5a8 is in m_tag_locate (/usr/src/sys/kern/uipc_mbuf2.c:389).
>> 384           if (t == NULL)
>> 385                   p = SLIST_FIRST(&m->m_pkthdr.tags);
>> 386           else
>> 387                   p = SLIST_NEXT(t, m_tag_link);
>> 388           while (p != NULL) {
>> 389                   if (p->m_tag_cookie == cookie && p->m_tag_id == type)
>> 390                           return p;
>> 391                   p = SLIST_NEXT(p, m_tag_link);
>> 392           }
>> 393           return NULL;
>> (kgdb) backtrace
>> #0  doadump () at pcpu.h:224
>> #1  0x805c25ce in boot (howto=260)
>>     at /usr/src/sys/kern/kern_shutdown.c:416
>> #2  0x805c29dc in panic (fmt=0x0)
>>     at /usr/src/sys/kern/kern_shutdown.c:590
>> #3  0x808d40bd in trap_fatal (frame=0x80c8af60,
>> eva=Variable "eva" is not available.
>> )
>>     at /usr/src/sys/amd64/amd64/trap.c:777
>> #4  0x808d4a8b in trap (frame=0xff8e6420)
>>     at /usr/src/sys/amd64/amd64/trap.c:588
>> #5  0x808b9d64 in calltrap ()
>>     at /usr/src/sys/amd64/amd64/exception.S:224
>> #6  0x8061f5a8 in m_tag_locate (m=0xff010bb45c00, cookie=0,
>>     type=6, t=Variable "t" is not available.
>> ) at /usr/src/sys/kern/uipc_mbuf2.c:388
>> #7  0x806d7

Re: Crashes on X7SPE-HF with em

[Philipp Wuensche]

In the end this turned out to be faulty hardware, at least the mainboard
died a tragic death and has to be replaced now.

Thanks for the help anyway and sorry for the noise!

Greetings,
philipp

Philipp Wuensche wrote:
> Hi,
> 
> I'm having trouble with a system on a Supermicro X7SPE-HF, it crashes
> about once a day. I haven't found a way to trigger this yet.
> 
> The system has a bunch of VLANs on em1, it does routing between them.
> 
> Currently its running 8-STABLE but it happend with 8.1-RELEASE too.
> 
> greetings,
> Philipp
> 
> # kgdb kernel.debug /var/crash/vmcore.0
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for details.
> This GDB was configured as "amd64-marcel-freebsd"...
> 
> Unread portion of the kernel message buffer:
> 
> 
> Fatal trap 9: general protection fault while in kernel mode
> cpuid = 0; apic id = 00
> instruction pointer   = 0x20:0x8061f5a8
> stack pointer = 0x28:0xff8e64d0
> frame pointer = 0x28:0xff8e64e0
> code segment  = base 0x0, limit 0xf, type 0x1b
>   = DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags  = interrupt enabled, resume, IOPL = 0
> current process   = 0 (em1 taskq)
> trap number   = 9
> panic: general protection fault
> cpuid = 0
> Uptime: 13h24m39s
> Physical memory: 4079 MB
> Dumping 1349 MB: 1334 1318 1302 1286 1270 1254 1238 1222 1206 1190 1174
> 1158 1142 1126 1110 1094 1078 1062 1046 1030 1014 998 982 966 950 934
> 918 902 886 870 854 838 822 806 790 774 758 742 726 710 694 678 662 646
> 630 614 598 582 566 550 534 518 502 486 470 454 438 422 406 390 374 358
> 342 326 310 294 278 262 246 230 214 198 182 166 150 134 118 102 86 70 54
> 38 22 6
> 
> Reading symbols from /boot/kernel/zfs.ko...Reading symbols from
> /boot/kernel/zfs.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/zfs.ko
> Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from
> /boot/kernel/opensolaris.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/opensolaris.ko
> Reading symbols from /boot/kernel/coretemp.ko...Reading symbols from
> /boot/kernel/coretemp.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/coretemp.ko
> Reading symbols from /boot/kernel/ahci.ko...Reading symbols from
> /boot/kernel/ahci.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/ahci.ko
> Reading symbols from /boot/kernel/ipmi.ko...Reading symbols from
> /boot/kernel/ipmi.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/ipmi.ko
> Reading symbols from /boot/kernel/smbus.ko...Reading symbols from
> /boot/kernel/smbus.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/smbus.ko
> Reading symbols from /boot/kernel/pflog.ko...Reading symbols from
> /boot/kernel/pflog.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/pflog.ko
> Reading symbols from /boot/kernel/pf.ko...Reading symbols from
> /boot/kernel/pf.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/pf.ko
> #0  doadump () at pcpu.h:224
> 224   __asm("movq %%gs:0,%0" : "=r" (td));
> (kgdb) list *0x8061f5a8
> 0x8061f5a8 is in m_tag_locate (/usr/src/sys/kern/uipc_mbuf2.c:389).
> 384   if (t == NULL)
> 385   p = SLIST_FIRST(&m->m_pkthdr.tags);
> 386   else
> 387   p = SLIST_NEXT(t, m_tag_link);
> 388   while (p != NULL) {
> 389   if (p->m_tag_cookie == cookie && p->m_tag_id == type)
> 390   return p;
> 391   p = SLIST_NEXT(p, m_tag_link);
> 392   }
> 393   return NULL;
> (kgdb) backtrace
> #0  doadump () at pcpu.h:224
> #1  0x805c25ce in boot (howto=260)
> at /usr/src/sys/kern/kern_shutdown.c:416
> #2  0x805c29dc in panic (fmt=0x0)
> at /usr/src/sys/kern/kern_shutdown.c:590
> #3  0x808d40bd in trap_fatal (frame=0x80c8af60,
> eva=Variable "eva" is not available.
> )
> at /usr/src/sys/amd64/amd64/trap.c:777
> #4  0x808d4a8b in trap (frame=0xff8e6420)
> at /usr/src/sys/amd64/amd64/trap.c:588
> #5  0x808b9d64 in calltrap ()
> at /usr/src/sys/amd64/amd64/exception.S:224
> #6  0x8061f5a8 in m_tag_locate (m=0xff010bb45c00, cookie=0,
> type=6, t=Variable "t" is not available.
> ) at /usr/src/sys/kern/uipc_mbuf2.c:388
> #7  0x806d7c56 in ip_ipsec_output (m=0xff8e6598,
> inp=0xff010be43150, flags=0xff8e6594,
> error=0xff8e65a8, ifp=Variable "ifp" is not available.
> ) at mbuf.h:1006
> #8  0x806d97ef in ip_output (m=0xff010bb45c00, opt=Variable
> "opt" is not available.
> )
> at /us

Re: Crashes on X7SPE-HF with em

[Greg Byshenk]

On Mon, Aug 30, 2010 at 04:08:45AM -0700, Jeremy Chadwick wrote:
> Bcc: 
> Subject: Re: igb related(?) panics on 7.3-STABLE
> Reply-To: 
> In-Reply-To: <20100830094631.gd12...@core.byshenk.net>
> 
> On Mon, Aug 30, 2010 at 11:46:31AM +0200, Greg Byshenk wrote:
> > On Sun, Aug 29, 2010 at 08:16:59PM +0200, Greg Byshenk wrote:
> > 
> > > I've begun seeing problems on a machine running FreeBSD-7.3-STABLE, 
> > > 64-bit,
> > > with two igb nics in use.  Previously the machine was fine, running 
> > > earlier
> > > versions of 7-STABLE, although the load on the network has increased due
> > > to additional machines being added to the network (the machine functions
> > > as a fileserver, serving files to compute machines via NFS(v3)).
> > > 
> > > Any advice is much appreciated. System info is below.
> > 
> > 
> > Followup with more information. The machine just panic'ed again, with 
> > a lot of load on the network.
> > 
> > Output from the 'systat' that was running at the time:
> > 
> >3 usersLoad 54.47 42.35 24.25  Aug 30 11:17
> > 
> >Mem:KBREALVIRTUAL   VN PAGER   SWAP 
> > PAGER
> >Tot   Share  TotShareFree   in   out in  
> >  out
> >Act   462325504   86814010548  943324  count
> >All  4564847852 1074772k27740  pages
> >Proc:
> > Interrupts
> >  r   p   d   s   w   Csw  Trp  Sys  Int  Sof  Fltcow   54220 
> > total
> >  1 170  392k8  278  22k  1951zfod
> > sio0 irq4
> >  ozfod   
> > fdc0 irq6
> >70.4%Sys   3.1%Intr  0.0%User  0.0%Nice 26.5%Idle%ozfod27 
> > twa0 uhci0
> >|||||||||||   daefr  2001 
> > cpu0: time
> >===++ prcfr   
> > igb0 256
> >  9938 dtbuf 1247 totfr   
> > igb0 257
> >Namei Name-cache   Dir-cache10 desvn  react   
> > igb0 258
> >   Callshits   %hits   % 34443 numvn1 pdwak   
> > igb0 259
> > 24996 frevn   112852 pdpgs   
> > igb0 262
> >  intrn   
> > igb0 263
> >Disks   da0   da1 pass0 pass1 2570672 wire
> > igb0 264
> >KB/t   0.00 12.23  0.00  0.00   46760 act 
> > igb0 265
> >tps   026 0 014706896 inact 19449 
> > igb1 266
> >MB/s   0.00  0.31  0.00  0.000 769796  26585
> >  021 0 0  173528
> > 
> > 
> > -greg
> >  
> >  
> >  
> > > Machine:
> > > ===
> > > 
> > > FreeBSD server.example.com 7.3-STABLE FreeBSD 7.3-STABLE #36: Wed Aug 25 
> > > 11:01:07 CEST 2010 
> > > r...@server.example.com:/usr/obj/usr/src/sys/KERNEL amd64
> > > 
> > > Kernel was csup'd earlier in the day on 25 August, immediately prior to 
> > > the build.
> > > 
> > > 
> > > Panic:
> > > ==
> > > 
> > > Fatal trap 9: general protection fault while in kernel mode
> > > cpuid = 2; apic id = 02
> > > instruction pointer = 0x8:0x8052f40c
> > > stack pointer   = 0x10:0xff82056819d0
> > > frame pointer   = 0x10:0xff82056819f0
> > > code segment= base 0x0, limit 0xf, type 0x1b
> > > = DPL 0, pres 1, long 1, def32 0, gran 1
> > > processor eflags= interrupt enabled, resume, IOPL = 0
> > > current process = 65 (igb1 que)
> > > trap number = 9
> > > panic: general protection fault
> > > cpuid = 2
> > > KDB: stack backtrace:
> > > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
> > > panic() at panic+0x182
> > > trap_fatal() at trap_fatal+0x294
> > > trap() at trap+0x106
> > > calltrap() at calltrap+0x8
> > > --- trap 0x9, rip = 0x8052f40c, rsp = 0xff82056819d0, rbp = 
> > > 0xff82056819f0 --- m_tag_delete_chain() at m_tag_delete_chain+0x1c
> > > uma_zfree_arg() at uma_zfree_arg+0x41
> > > m_freem() at m_freem+0x54
> > > ether_demux() at ether_demux+0x85
> > > ether_input() at ether_input+0x1bb
> > > igb_rxeof() at igb_rxeof+0x29d
> > > igb_handle_que() at igb_handle_que+0x9a
> > > taskqueue_run() at taskqueue_run+0xac
> > > taskqueue_thread_loop() at taskqueue_thread_loop+0x46
> > > fork_exit() at fork_exit+0x122
> > > fork_trampoline() at fork_trampoline+0xe
> > > --- trap 0, rip = 0, rsp = 0xff8205681d30, rbp = 0 ---
> > > Uptime: 11h57m6s
> > > Physical memory: 18411 MB
> > > Dumping 3770 MB:
> > > 
> > > Fatal trap 12: page fault while in kernel mode
> > > cpuid = 0; apic id = 00
> > > fault virtual address   = 0x80
> > > fault code  = supervis

Re: Crashes on X7SPE-HF with em

[Jeremy Chadwick]

Bcc: 
Subject: Re: igb related(?) panics on 7.3-STABLE
Reply-To: 
In-Reply-To: <20100830094631.gd12...@core.byshenk.net>

On Mon, Aug 30, 2010 at 11:46:31AM +0200, Greg Byshenk wrote:
> On Sun, Aug 29, 2010 at 08:16:59PM +0200, Greg Byshenk wrote:
> 
> > I've begun seeing problems on a machine running FreeBSD-7.3-STABLE, 64-bit,
> > with two igb nics in use.  Previously the machine was fine, running earlier
> > versions of 7-STABLE, although the load on the network has increased due
> > to additional machines being added to the network (the machine functions
> > as a fileserver, serving files to compute machines via NFS(v3)).
> > 
> > Any advice is much appreciated. System info is below.
> 
> 
> Followup with more information. The machine just panic'ed again, with 
> a lot of load on the network.
> 
> Output from the 'systat' that was running at the time:
> 
>3 usersLoad 54.47 42.35 24.25  Aug 30 11:17
> 
>Mem:KBREALVIRTUAL   VN PAGER   SWAP 
> PAGER
>Tot   Share  TotShareFree   in   out in   
> out
>Act   462325504   86814010548  943324  count
>All  4564847852 1074772k27740  pages
>Proc:Interrupts
>  r   p   d   s   w   Csw  Trp  Sys  Int  Sof  Fltcow   54220 total
>  1 170  392k8  278  22k  1951zfodsio0 
> irq4
>  ozfod   fdc0 
> irq6
>70.4%Sys   3.1%Intr  0.0%User  0.0%Nice 26.5%Idle%ozfod27 twa0 
> uhci0
>|||||||||||   daefr  2001 
> cpu0: time
>===++ prcfr   igb0 
> 256
>  9938 dtbuf 1247 totfr   igb0 
> 257
>Namei Name-cache   Dir-cache10 desvn  react   igb0 
> 258
>   Callshits   %hits   % 34443 numvn1 pdwak   igb0 
> 259
> 24996 frevn   112852 pdpgs   igb0 
> 262
>  intrn   igb0 
> 263
>Disks   da0   da1 pass0 pass1 2570672 wireigb0 
> 264
>KB/t   0.00 12.23  0.00  0.00   46760 act igb0 
> 265
>tps   026 0 014706896 inact 19449 igb1 
> 266
>MB/s   0.00  0.31  0.00  0.000 769796  26585
>  021 0 0  173528
> 
> 
> -greg
>  
>  
>  
> > Machine:
> > ===
> > 
> > FreeBSD server.example.com 7.3-STABLE FreeBSD 7.3-STABLE #36: Wed Aug 25 
> > 11:01:07 CEST 2010 r...@server.example.com:/usr/obj/usr/src/sys/KERNEL 
> > amd64
> > 
> > Kernel was csup'd earlier in the day on 25 August, immediately prior to 
> > the build.
> > 
> > 
> > Panic:
> > ==
> > 
> > Fatal trap 9: general protection fault while in kernel mode
> > cpuid = 2; apic id = 02
> > instruction pointer = 0x8:0x8052f40c
> > stack pointer   = 0x10:0xff82056819d0
> > frame pointer   = 0x10:0xff82056819f0
> > code segment= base 0x0, limit 0xf, type 0x1b
> > = DPL 0, pres 1, long 1, def32 0, gran 1
> > processor eflags= interrupt enabled, resume, IOPL = 0
> > current process = 65 (igb1 que)
> > trap number = 9
> > panic: general protection fault
> > cpuid = 2
> > KDB: stack backtrace:
> > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a
> > panic() at panic+0x182
> > trap_fatal() at trap_fatal+0x294
> > trap() at trap+0x106
> > calltrap() at calltrap+0x8
> > --- trap 0x9, rip = 0x8052f40c, rsp = 0xff82056819d0, rbp = 
> > 0xff82056819f0 --- m_tag_delete_chain() at m_tag_delete_chain+0x1c
> > uma_zfree_arg() at uma_zfree_arg+0x41
> > m_freem() at m_freem+0x54
> > ether_demux() at ether_demux+0x85
> > ether_input() at ether_input+0x1bb
> > igb_rxeof() at igb_rxeof+0x29d
> > igb_handle_que() at igb_handle_que+0x9a
> > taskqueue_run() at taskqueue_run+0xac
> > taskqueue_thread_loop() at taskqueue_thread_loop+0x46
> > fork_exit() at fork_exit+0x122
> > fork_trampoline() at fork_trampoline+0xe
> > --- trap 0, rip = 0, rsp = 0xff8205681d30, rbp = 0 ---
> > Uptime: 11h57m6s
> > Physical memory: 18411 MB
> > Dumping 3770 MB:
> > 
> > Fatal trap 12: page fault while in kernel mode
> > cpuid = 0; apic id = 00
> > fault virtual address   = 0x80
> > fault code  = supervisor write data, page not present
> > instruction pointer = 0x8:0x80188b5f
> > stack pointer   = 0x10:0xff82056811f0
> > frame pointer   = 0x10:0xff82056812f0
> > code segment= base 0x0, limit 0xf, type 0x1b
> > = DPL 0, pres 1, long 1, def32 0, gran 1

Re: Crashes on X7SPE-HF with em

[Pyun YongHyeon]

On Sat, Aug 28, 2010 at 04:19:07PM +0200, Philipp Wuensche wrote:
> Philipp Wuensche wrote:
> > 
> > It just now started running the kernel without IPSEC and ALTQ.
> 
> Here we go again, this time it crashed with IPSEC and ALTQ disabled,
> crashdump looks different this time though.
> 
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for details.
> This GDB was configured as "amd64-marcel-freebsd"...
> 
> Unread portion of the kernel message buffer:
> 
> 
> Fatal trap 12: page fault while in kernel mode
> cpuid = 0; apic id = 00
> fault virtual address = 0x80400bc58038
> fault code= supervisor read data, page not present
> instruction pointer   = 0x20:0x808a41ae
> stack pointer = 0x28:0xff8e69a0
> frame pointer = 0x28:0xff8e69b0
> code segment  = base 0x0, limit 0xf, type 0x1b
>   = DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags  = interrupt enabled, resume, IOPL = 0
> current process   = 0 (em1 taskq)
> trap number   = 12
> panic: page fault
> cpuid = 0
> Uptime: 23h30m3s
> Physical memory: 4079 MB
> Dumping 1907 MB: 1892 1876em1: Watchdog timeout -- resetting
>  1860 1844 1828 1812 1796 1780 1764 1748 1732 1716 1700 1684 1668 1652
> 1636 1620 1604 1588 1572 1556 1540 1524 1508 1492 1476 1460 1444 1428
> 1412 1396 1380 1364 1348 1332 1316 1300 1284 1268 1252 1236 1220 1204
> 1188 1172 1156 1140 1124 1108 1092 1076 1060 1044 1028 1012 996 980 964
> 948 932 916 900 884 868 852 836 820 804 788 772 756 740 724 708 692 676
> 660 644 628 612 596 580 564 548 532 516 500 484 468 452 436 420 404 388
> 372 356 340 324 308 292 276 260 244 228 212 196 180 164 148 132 116 100
> 84 68 52 36 20 4
> 
> Reading symbols from /boot/kernel/zfs.ko...Reading symbols from
> /boot/kernel/zfs.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/zfs.ko
> Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from
> /boot/kernel/opensolaris.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/opensolaris.ko
> Reading symbols from /boot/kernel/geom_stripe.ko...Reading symbols from
> /boot/kernel/geom_stripe.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/geom_stripe.ko
> Reading symbols from /boot/kernel/coretemp.ko...Reading symbols from
> /boot/kernel/coretemp.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/coretemp.ko
> Reading symbols from /boot/kernel/ahci.ko...Reading symbols from
> /boot/kernel/ahci.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/ahci.ko
> Reading symbols from /boot/kernel/ipmi.ko...Reading symbols from
> /boot/kernel/ipmi.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/ipmi.ko
> Reading symbols from /boot/kernel/smbus.ko...Reading symbols from
> /boot/kernel/smbus.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/smbus.ko
> Reading symbols from /boot/kernel/pflog.ko...Reading symbols from
> /boot/kernel/pflog.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/pflog.ko
> Reading symbols from /boot/kernel/pf.ko...Reading symbols from
> /boot/kernel/pf.ko.symbols...done.
> done.
> Loaded symbols for /boot/kernel/pf.ko
> #0  doadump () at pcpu.h:224
> 224   __asm("movq %%gs:0,%0" : "=r" (td));
> (kgdb) list *0x808a41ae
> 0x808a41ae is in pmap_kextract
> (/usr/src/sys/amd64/amd64/pmap.c:1172).
> 1167  vm_paddr_t pa;
> 1168  
> 1169  if (va >= DMAP_MIN_ADDRESS && va < DMAP_MAX_ADDRESS) {
> 1170  pa = DMAP_TO_PHYS(va);
> 1171  } else {
> 1172  pde = *vtopde(va);
> 1173  if (pde & PG_PS) {
> 1174  pa = (pde & PG_PS_FRAME) | (va & PDRMASK);
> 1175  } else {
> 1176  /*
> (kgdb) backtrace
> #0  doadump () at pcpu.h:224
> #1  0x805b2b5e in boot (howto=260)
> at /usr/src/sys/kern/kern_shutdown.c:416
> #2  0x805b2f6c in panic (fmt=0x0)
> at /usr/src/sys/kern/kern_shutdown.c:590
> #3  0x808ac70d in trap_fatal (frame=0x80c5cc60,
> eva=Variable "eva" is not available.
> )
> at /usr/src/sys/amd64/amd64/trap.c:777
> #4  0x808acacf in trap_pfault (frame=0xff8e68f0, usermode=0)
> at /usr/src/sys/amd64/amd64/trap.c:693
> #5  0x808ad2e2 in trap (frame=0xff8e68f0)
> at /usr/src/sys/amd64/amd64/trap.c:451
> #6  0x808923b4 in calltrap ()
> at /usr/src/sys/amd64/amd64/exception.S:224
> #7  0x808a41ae in pmap_kextract (va=51771551252551)
> at /usr/src/sys/amd64/amd64/pmap.c:1172
> #8  0x80890f83 in bus_dmamap_load_mbuf_sg (dmat=0xff0002727c00,
> m

Re: Crashes on X7SPE-HF with em

[Philipp Wuensche]

Philipp Wuensche wrote:
> 
> It just now started running the kernel without IPSEC and ALTQ.

Here we go again, this time it crashed with IPSEC and ALTQ disabled,
crashdump looks different this time though.

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x80400bc58038
fault code  = supervisor read data, page not present
instruction pointer = 0x20:0x808a41ae
stack pointer   = 0x28:0xff8e69a0
frame pointer   = 0x28:0xff8e69b0
code segment= base 0x0, limit 0xf, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags= interrupt enabled, resume, IOPL = 0
current process = 0 (em1 taskq)
trap number = 12
panic: page fault
cpuid = 0
Uptime: 23h30m3s
Physical memory: 4079 MB
Dumping 1907 MB: 1892 1876em1: Watchdog timeout -- resetting
 1860 1844 1828 1812 1796 1780 1764 1748 1732 1716 1700 1684 1668 1652
1636 1620 1604 1588 1572 1556 1540 1524 1508 1492 1476 1460 1444 1428
1412 1396 1380 1364 1348 1332 1316 1300 1284 1268 1252 1236 1220 1204
1188 1172 1156 1140 1124 1108 1092 1076 1060 1044 1028 1012 996 980 964
948 932 916 900 884 868 852 836 820 804 788 772 756 740 724 708 692 676
660 644 628 612 596 580 564 548 532 516 500 484 468 452 436 420 404 388
372 356 340 324 308 292 276 260 244 228 212 196 180 164 148 132 116 100
84 68 52 36 20 4

Reading symbols from /boot/kernel/zfs.ko...Reading symbols from
/boot/kernel/zfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from
/boot/kernel/opensolaris.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
Reading symbols from /boot/kernel/geom_stripe.ko...Reading symbols from
/boot/kernel/geom_stripe.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/geom_stripe.ko
Reading symbols from /boot/kernel/coretemp.ko...Reading symbols from
/boot/kernel/coretemp.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/coretemp.ko
Reading symbols from /boot/kernel/ahci.ko...Reading symbols from
/boot/kernel/ahci.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ahci.ko
Reading symbols from /boot/kernel/ipmi.ko...Reading symbols from
/boot/kernel/ipmi.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ipmi.ko
Reading symbols from /boot/kernel/smbus.ko...Reading symbols from
/boot/kernel/smbus.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/smbus.ko
Reading symbols from /boot/kernel/pflog.ko...Reading symbols from
/boot/kernel/pflog.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/pflog.ko
Reading symbols from /boot/kernel/pf.ko...Reading symbols from
/boot/kernel/pf.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/pf.ko
#0  doadump () at pcpu.h:224
224 __asm("movq %%gs:0,%0" : "=r" (td));
(kgdb) list *0x808a41ae
0x808a41ae is in pmap_kextract
(/usr/src/sys/amd64/amd64/pmap.c:1172).
1167vm_paddr_t pa;
1168
1169if (va >= DMAP_MIN_ADDRESS && va < DMAP_MAX_ADDRESS) {
1170pa = DMAP_TO_PHYS(va);
1171} else {
1172pde = *vtopde(va);
1173if (pde & PG_PS) {
1174pa = (pde & PG_PS_FRAME) | (va & PDRMASK);
1175} else {
1176/*
(kgdb) backtrace
#0  doadump () at pcpu.h:224
#1  0x805b2b5e in boot (howto=260)
at /usr/src/sys/kern/kern_shutdown.c:416
#2  0x805b2f6c in panic (fmt=0x0)
at /usr/src/sys/kern/kern_shutdown.c:590
#3  0x808ac70d in trap_fatal (frame=0x80c5cc60,
eva=Variable "eva" is not available.
)
at /usr/src/sys/amd64/amd64/trap.c:777
#4  0x808acacf in trap_pfault (frame=0xff8e68f0, usermode=0)
at /usr/src/sys/amd64/amd64/trap.c:693
#5  0x808ad2e2 in trap (frame=0xff8e68f0)
at /usr/src/sys/amd64/amd64/trap.c:451
#6  0x808923b4 in calltrap ()
at /usr/src/sys/amd64/amd64/exception.S:224
#7  0x808a41ae in pmap_kextract (va=51771551252551)
at /usr/src/sys/amd64/amd64/pmap.c:1172
#8  0x80890f83 in bus_dmamap_load_mbuf_sg (dmat=0xff0002727c00,
map=0x80c99d40, m0=Variable "m0" is not available.
)
at /usr/src/sys/amd64/amd64/busdma_machdep.c:659
#9  0x8032f8fc in em_refresh_mbufs (rxr=0xff0002712600,
limit=975)
at /usr/src/sys/dev/e1000/if_em.c:3691
#10 0x8032ff3c in em_

Re: Crashes on X7SPE-HF with em

[Philipp Wuensche]

Jack Vogel wrote:
> 
> 
> On Fri, Aug 27, 2010 at 6:04 AM, Philipp Wuensche  > wrote:
> 
> Jack Vogel wrote:
> >
> >
> > On Thu, Aug 26, 2010 at 3:15 PM, Jeremy Chadwick
> > mailto:free...@jdc.parodius.com>
> >>
> wrote:
> >
> > On Thu, Aug 26, 2010 at 11:56:48PM +0200, Philipp Wuensche wrote:
> > > Jeremy Chadwick wrote:
> > > >
> > > > CC'ing Jack Vogel of Intel and Yong-Hyeon PYUN who might
> have some
> > > > ideas.  OP's backtrace is here:
> > > >
> > > >
> >
> http://lists.freebsd.org/pipermail/freebsd-stable/2010-August/058425.html
> > > >
> > > > Philipp, can you please provide the following output?
> > > >
> > > > * dmesg | egrep 'em[0-9]'
> > >
> > > em0:  port
> > 0xdc00-0xdc1f mem
> > > 0xfe9e-0xfe9f,0xfe9dc000-0xfe9d irq 16 at device 0.0
> > on pci2
> > > em0: Using MSI interrupt
> > > em0: [FILTER]
> > > em0: Ethernet address: 00:25:90:04:6e:fa
> > > em1:  port
> > 0xec00-0xec1f mem
> > > 0xfeae-0xfeaf,0xfeadc000-0xfead irq 17 at device 0.0
> > on pci3
> > > em1: Using MSI interrupt
> > > em1: [FILTER]
> > > em1: Ethernet address: 00:25:90:04:6e:fb
> > >
> > > > * uname -a (you can XXX out the machine name if
> need be)
> > >
> > > FreeBSD XXX 8.1-STABLE FreeBSD 8.1-STABLE #2: Wed Aug 25
> 10:38:50 CEST
> > > 2010 r...@xxx:/usr/obj/usr/src/sys/XXX  amd64
> > >
> > > Date of source is Aug 17 14:09 CEST 2010. It happend with
> 8.1-RELEASE
> > > too, I can go back to RELEASE or any SVN revision you would
> like,
> > if it
> > > is helping in any way.
> > >
> > > Kernel-config:
> > >
> > > include GENERIC
> > >
> > > ident   XXX
> > >
> > > options IPSEC
> > >
> > > options   DEVICE_POLLING
> > > options ACCEPT_FILTER_HTTP
> > >
> > > options ALTQ
> > >
> > > options ALTQ_CBQ
> > > options ALTQ_RED
> > > options ALTQ_RIO
> > > options ALTQ_HFSC
> > > options ALTQ_PRIQ
> > >
> > > devicecrypto
> > > deviceenc
> > >
> > >
> > > > * pciconf -lvc (only include the em(4) items please)
> > >
> > > e...@pci0:2:0:0:   class=0x02 card=0x060a15d9
> > chip=0x10d38086 rev=0x00
> > > hdr=0x00
> > > vendor = 'Intel Corporation'
> > > device = 'Intel 82574L Gigabit Ethernet Controller
> (82574L)'
> > > class  = network
> > > subclass   = ethernet
> > > cap 01[c8] = powerspec 2  supports D0 D3  current D0
> > > cap 05[d0] = MSI supports 1 message, 64 bit enabled with
> 1 message
> > > cap 10[e0] = PCI-Express 1 endpoint max data 128(256)
> link x1(x1)
> > > cap 11[a0] = MSI-X supports 5 messages in map 0x1c
> > > e...@pci0:3:0:0:   class=0x02 card=0x060a15d9
> > chip=0x10d38086 rev=0x00
> > > hdr=0x00
> > > vendor = 'Intel Corporation'
> > > device = 'Intel 82574L Gigabit Ethernet Controller
> (82574L)'
> > > class  = network
> > > subclass   = ethernet
> > > cap 01[c8] = powerspec 2  supports D0 D3  current D0
> > > cap 05[d0] = MSI supports 1 message, 64 bit enabled with
> 1 message
> > > cap 10[e0] = PCI-Express 1 endpoint max data 128(256)
> link x1(x1)
> > > cap 11[a0] = MSI-X supports 5 messages in map 0x1c
> > >
> > > > * vmstat -i
> > >
> > > interrupt  total   rate
> > > irq1: atkbd0   9  0
> > > cpu0: timer 36544552   1994
> > > irq256: em0 3801  0
> > > irq257: em1 32963909   1799
> > > irq258: ahci0 175662  9
> > > cpu1: timer 36543525   1994
> > > cpu2: timer 36543525   1994
> > > cpu3: timer 36543525   1994
> > > Total  179318508   9786
> > >
> > > There is an shared IPMI interface on em0, but the interface
> is not
> > used
> > > by FreeBSD. em1 is used by four VLANs. Polling is only in the
> > > Kernelconfig, not activated on the devices.
> >

Re: Crashes on X7SPE-HF with em

[Jack Vogel]

On Fri, Aug 27, 2010 at 6:04 AM, Philipp Wuensche wrote:

> Jack Vogel wrote:
> >
> >
> > On Thu, Aug 26, 2010 at 3:15 PM, Jeremy Chadwick
> > mailto:free...@jdc.parodius.com>> wrote:
> >
> > On Thu, Aug 26, 2010 at 11:56:48PM +0200, Philipp Wuensche wrote:
> > > Jeremy Chadwick wrote:
> > > >
> > > > CC'ing Jack Vogel of Intel and Yong-Hyeon PYUN who might have
> some
> > > > ideas.  OP's backtrace is here:
> > > >
> > > >
> >
> http://lists.freebsd.org/pipermail/freebsd-stable/2010-August/058425.html
> > > >
> > > > Philipp, can you please provide the following output?
> > > >
> > > > * dmesg | egrep 'em[0-9]'
> > >
> > > em0:  port
> > 0xdc00-0xdc1f mem
> > > 0xfe9e-0xfe9f,0xfe9dc000-0xfe9d irq 16 at device 0.0
> > on pci2
> > > em0: Using MSI interrupt
> > > em0: [FILTER]
> > > em0: Ethernet address: 00:25:90:04:6e:fa
> > > em1:  port
> > 0xec00-0xec1f mem
> > > 0xfeae-0xfeaf,0xfeadc000-0xfead irq 17 at device 0.0
> > on pci3
> > > em1: Using MSI interrupt
> > > em1: [FILTER]
> > > em1: Ethernet address: 00:25:90:04:6e:fb
> > >
> > > > * uname -a (you can XXX out the machine name if need be)
> > >
> > > FreeBSD XXX 8.1-STABLE FreeBSD 8.1-STABLE #2: Wed Aug 25 10:38:50
> CEST
> > > 2010 r...@xxx:/usr/obj/usr/src/sys/XXX  amd64
> > >
> > > Date of source is Aug 17 14:09 CEST 2010. It happend with
> 8.1-RELEASE
> > > too, I can go back to RELEASE or any SVN revision you would like,
> > if it
> > > is helping in any way.
> > >
> > > Kernel-config:
> > >
> > > include GENERIC
> > >
> > > ident   XXX
> > >
> > > options IPSEC
> > >
> > > options   DEVICE_POLLING
> > > options ACCEPT_FILTER_HTTP
> > >
> > > options ALTQ
> > >
> > > options ALTQ_CBQ
> > > options ALTQ_RED
> > > options ALTQ_RIO
> > > options ALTQ_HFSC
> > > options ALTQ_PRIQ
> > >
> > > devicecrypto
> > > deviceenc
> > >
> > >
> > > > * pciconf -lvc (only include the em(4) items please)
> > >
> > > e...@pci0:2:0:0:   class=0x02 card=0x060a15d9
> > chip=0x10d38086 rev=0x00
> > > hdr=0x00
> > > vendor = 'Intel Corporation'
> > > device = 'Intel 82574L Gigabit Ethernet Controller
> (82574L)'
> > > class  = network
> > > subclass   = ethernet
> > > cap 01[c8] = powerspec 2  supports D0 D3  current D0
> > > cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1
> message
> > > cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link
> x1(x1)
> > > cap 11[a0] = MSI-X supports 5 messages in map 0x1c
> > > e...@pci0:3:0:0:   class=0x02 card=0x060a15d9
> > chip=0x10d38086 rev=0x00
> > > hdr=0x00
> > > vendor = 'Intel Corporation'
> > > device = 'Intel 82574L Gigabit Ethernet Controller
> (82574L)'
> > > class  = network
> > > subclass   = ethernet
> > > cap 01[c8] = powerspec 2  supports D0 D3  current D0
> > > cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1
> message
> > > cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link
> x1(x1)
> > > cap 11[a0] = MSI-X supports 5 messages in map 0x1c
> > >
> > > > * vmstat -i
> > >
> > > interrupt  total   rate
> > > irq1: atkbd0   9  0
> > > cpu0: timer 36544552   1994
> > > irq256: em0 3801  0
> > > irq257: em1 32963909   1799
> > > irq258: ahci0 175662  9
> > > cpu1: timer 36543525   1994
> > > cpu2: timer 36543525   1994
> > > cpu3: timer 36543525   1994
> > > Total  179318508   9786
> > >
> > > There is an shared IPMI interface on em0, but the interface is not
> > used
> > > by FreeBSD. em1 is used by four VLANs. Polling is only in the
> > > Kernelconfig, not activated on the devices.
> >
> > So much complexity here.  Tracking this down might be difficult.
> >
> > One thing that does concern me is the interrupt rate for em1.  Jack
> et
> > al, is this normal?  I don't see this behaviour on my 8.x systems
> with
> > em(4) driver 7.0.5, but my systems all use 82573E and 82573L, and
> don't
> > have MSI-X support.
> >
> >
> > He is only using one vector anyway it seems, so MSIX isnt making things
> > much more complex than your 573.
> >
> > The interrupt rate seems high but I'm not sure if its abnormal for a busy
> > interface.
> >
> > I tend to agree with Yongari, let's eliminate all the complicating

Re: Crashes on X7SPE-HF with em

[Philipp Wuensche]

Jack Vogel wrote:
> 
> 
> On Thu, Aug 26, 2010 at 3:15 PM, Jeremy Chadwick
> mailto:free...@jdc.parodius.com>> wrote:
> 
> On Thu, Aug 26, 2010 at 11:56:48PM +0200, Philipp Wuensche wrote:
> > Jeremy Chadwick wrote:
> > >
> > > CC'ing Jack Vogel of Intel and Yong-Hyeon PYUN who might have some
> > > ideas.  OP's backtrace is here:
> > >
> > >
> http://lists.freebsd.org/pipermail/freebsd-stable/2010-August/058425.html
> > >
> > > Philipp, can you please provide the following output?
> > >
> > > * dmesg | egrep 'em[0-9]'
> >
> > em0:  port
> 0xdc00-0xdc1f mem
> > 0xfe9e-0xfe9f,0xfe9dc000-0xfe9d irq 16 at device 0.0
> on pci2
> > em0: Using MSI interrupt
> > em0: [FILTER]
> > em0: Ethernet address: 00:25:90:04:6e:fa
> > em1:  port
> 0xec00-0xec1f mem
> > 0xfeae-0xfeaf,0xfeadc000-0xfead irq 17 at device 0.0
> on pci3
> > em1: Using MSI interrupt
> > em1: [FILTER]
> > em1: Ethernet address: 00:25:90:04:6e:fb
> >
> > > * uname -a (you can XXX out the machine name if need be)
> >
> > FreeBSD XXX 8.1-STABLE FreeBSD 8.1-STABLE #2: Wed Aug 25 10:38:50 CEST
> > 2010 r...@xxx:/usr/obj/usr/src/sys/XXX  amd64
> >
> > Date of source is Aug 17 14:09 CEST 2010. It happend with 8.1-RELEASE
> > too, I can go back to RELEASE or any SVN revision you would like,
> if it
> > is helping in any way.
> >
> > Kernel-config:
> >
> > include GENERIC
> >
> > ident   XXX
> >
> > options IPSEC
> >
> > options   DEVICE_POLLING
> > options ACCEPT_FILTER_HTTP
> >
> > options ALTQ
> >
> > options ALTQ_CBQ
> > options ALTQ_RED
> > options ALTQ_RIO
> > options ALTQ_HFSC
> > options ALTQ_PRIQ
> >
> > devicecrypto
> > deviceenc
> >
> >
> > > * pciconf -lvc (only include the em(4) items please)
> >
> > e...@pci0:2:0:0:   class=0x02 card=0x060a15d9
> chip=0x10d38086 rev=0x00
> > hdr=0x00
> > vendor = 'Intel Corporation'
> > device = 'Intel 82574L Gigabit Ethernet Controller (82574L)'
> > class  = network
> > subclass   = ethernet
> > cap 01[c8] = powerspec 2  supports D0 D3  current D0
> > cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
> > cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1)
> > cap 11[a0] = MSI-X supports 5 messages in map 0x1c
> > e...@pci0:3:0:0:   class=0x02 card=0x060a15d9
> chip=0x10d38086 rev=0x00
> > hdr=0x00
> > vendor = 'Intel Corporation'
> > device = 'Intel 82574L Gigabit Ethernet Controller (82574L)'
> > class  = network
> > subclass   = ethernet
> > cap 01[c8] = powerspec 2  supports D0 D3  current D0
> > cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
> > cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1)
> > cap 11[a0] = MSI-X supports 5 messages in map 0x1c
> >
> > > * vmstat -i
> >
> > interrupt  total   rate
> > irq1: atkbd0   9  0
> > cpu0: timer 36544552   1994
> > irq256: em0 3801  0
> > irq257: em1 32963909   1799
> > irq258: ahci0 175662  9
> > cpu1: timer 36543525   1994
> > cpu2: timer 36543525   1994
> > cpu3: timer 36543525   1994
> > Total  179318508   9786
> >
> > There is an shared IPMI interface on em0, but the interface is not
> used
> > by FreeBSD. em1 is used by four VLANs. Polling is only in the
> > Kernelconfig, not activated on the devices.
> 
> So much complexity here.  Tracking this down might be difficult.
> 
> One thing that does concern me is the interrupt rate for em1.  Jack et
> al, is this normal?  I don't see this behaviour on my 8.x systems with
> em(4) driver 7.0.5, but my systems all use 82573E and 82573L, and don't
> have MSI-X support.
> 
> 
> He is only using one vector anyway it seems, so MSIX isnt making things
> much more complex than your 573.
> 
> The interrupt rate seems high but I'm not sure if its abnormal for a busy
> interface.
> 
> I tend to agree with Yongari, let's eliminate all the complicating factors
> like IPSEC and ALTQ and see if it still occurs.

Crashed couply of minutes ago, the kernel without ALTQ and IPSEC is now
running.

greetings,
philipp
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe,

Re: Crashes on X7SPE-HF with em

[Jack Vogel]

On Thu, Aug 26, 2010 at 3:15 PM, Jeremy Chadwick
wrote:

> On Thu, Aug 26, 2010 at 11:56:48PM +0200, Philipp Wuensche wrote:
> > Jeremy Chadwick wrote:
> > >
> > > CC'ing Jack Vogel of Intel and Yong-Hyeon PYUN who might have some
> > > ideas.  OP's backtrace is here:
> > >
> > >
> http://lists.freebsd.org/pipermail/freebsd-stable/2010-August/058425.html
> > >
> > > Philipp, can you please provide the following output?
> > >
> > > * dmesg | egrep 'em[0-9]'
> >
> > em0:  port 0xdc00-0xdc1f mem
> > 0xfe9e-0xfe9f,0xfe9dc000-0xfe9d irq 16 at device 0.0 on pci2
> > em0: Using MSI interrupt
> > em0: [FILTER]
> > em0: Ethernet address: 00:25:90:04:6e:fa
> > em1:  port 0xec00-0xec1f mem
> > 0xfeae-0xfeaf,0xfeadc000-0xfead irq 17 at device 0.0 on pci3
> > em1: Using MSI interrupt
> > em1: [FILTER]
> > em1: Ethernet address: 00:25:90:04:6e:fb
> >
> > > * uname -a (you can XXX out the machine name if need be)
> >
> > FreeBSD XXX 8.1-STABLE FreeBSD 8.1-STABLE #2: Wed Aug 25 10:38:50 CEST
> > 2010 r...@xxx:/usr/obj/usr/src/sys/XXX  amd64
> >
> > Date of source is Aug 17 14:09 CEST 2010. It happend with 8.1-RELEASE
> > too, I can go back to RELEASE or any SVN revision you would like, if it
> > is helping in any way.
> >
> > Kernel-config:
> >
> > include GENERIC
> >
> > ident   XXX
> >
> > options IPSEC
> >
> > options   DEVICE_POLLING
> > options ACCEPT_FILTER_HTTP
> >
> > options ALTQ
> >
> > options ALTQ_CBQ
> > options ALTQ_RED
> > options ALTQ_RIO
> > options ALTQ_HFSC
> > options ALTQ_PRIQ
> >
> > devicecrypto
> > deviceenc
> >
> >
> > > * pciconf -lvc (only include the em(4) items please)
> >
> > e...@pci0:2:0:0:   class=0x02 card=0x060a15d9 chip=0x10d38086
> rev=0x00
> > hdr=0x00
> > vendor = 'Intel Corporation'
> > device = 'Intel 82574L Gigabit Ethernet Controller (82574L)'
> > class  = network
> > subclass   = ethernet
> > cap 01[c8] = powerspec 2  supports D0 D3  current D0
> > cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
> > cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1)
> > cap 11[a0] = MSI-X supports 5 messages in map 0x1c
> > e...@pci0:3:0:0:   class=0x02 card=0x060a15d9 chip=0x10d38086
> rev=0x00
> > hdr=0x00
> > vendor = 'Intel Corporation'
> > device = 'Intel 82574L Gigabit Ethernet Controller (82574L)'
> > class  = network
> > subclass   = ethernet
> > cap 01[c8] = powerspec 2  supports D0 D3  current D0
> > cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
> > cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1)
> > cap 11[a0] = MSI-X supports 5 messages in map 0x1c
> >
> > > * vmstat -i
> >
> > interrupt  total   rate
> > irq1: atkbd0   9  0
> > cpu0: timer 36544552   1994
> > irq256: em0 3801  0
> > irq257: em1 32963909   1799
> > irq258: ahci0 175662  9
> > cpu1: timer 36543525   1994
> > cpu2: timer 36543525   1994
> > cpu3: timer 36543525   1994
> > Total  179318508   9786
> >
> > There is an shared IPMI interface on em0, but the interface is not used
> > by FreeBSD. em1 is used by four VLANs. Polling is only in the
> > Kernelconfig, not activated on the devices.
>
> So much complexity here.  Tracking this down might be difficult.
>
> One thing that does concern me is the interrupt rate for em1.  Jack et
> al, is this normal?  I don't see this behaviour on my 8.x systems with
> em(4) driver 7.0.5, but my systems all use 82573E and 82573L, and don't
> have MSI-X support.
>
>
He is only using one vector anyway it seems, so MSIX isnt making things
much more complex than your 573.

The interrupt rate seems high but I'm not sure if its abnormal for a busy
interface.

I tend to agree with Yongari, let's eliminate all the complicating factors
like IPSEC and ALTQ and see if it still occurs.

>From the crash data I do not see a clear cause either.

Jack
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Crashes on X7SPE-HF with em

[Pyun YongHyeon]

On Fri, Aug 27, 2010 at 12:28:14AM +0200, Philipp Wuensche wrote:
> Jeremy Chadwick wrote:
> > On Thu, Aug 26, 2010 at 11:56:48PM +0200, Philipp Wuensche wrote:
> >> Jeremy Chadwick wrote:
> >>> CC'ing Jack Vogel of Intel and Yong-Hyeon PYUN who might have some
> >>> ideas.  OP's backtrace is here:
> >>>
> >>> http://lists.freebsd.org/pipermail/freebsd-stable/2010-August/058425.html
> >>>
> >>> Philipp, can you please provide the following output?
> >>>
> >>> * dmesg | egrep 'em[0-9]'
> >> em0:  port 0xdc00-0xdc1f mem
> >> 0xfe9e-0xfe9f,0xfe9dc000-0xfe9d irq 16 at device 0.0 on pci2
> >> em0: Using MSI interrupt
> >> em0: [FILTER]
> >> em0: Ethernet address: 00:25:90:04:6e:fa
> >> em1:  port 0xec00-0xec1f mem
> >> 0xfeae-0xfeaf,0xfeadc000-0xfead irq 17 at device 0.0 on pci3
> >> em1: Using MSI interrupt
> >> em1: [FILTER]
> >> em1: Ethernet address: 00:25:90:04:6e:fb
> >>
> >>> * uname -a (you can XXX out the machine name if need be)
> >> FreeBSD XXX 8.1-STABLE FreeBSD 8.1-STABLE #2: Wed Aug 25 10:38:50 CEST
> >> 2010 r...@xxx:/usr/obj/usr/src/sys/XXX  amd64
> >>
> >> Date of source is Aug 17 14:09 CEST 2010. It happend with 8.1-RELEASE
> >> too, I can go back to RELEASE or any SVN revision you would like, if it
> >> is helping in any way.
> >>
> >> Kernel-config:
> >>
> >> include GENERIC
> >>
> >> ident   XXX
> >>
> >> options IPSEC
> >>
> >> optionsDEVICE_POLLING
> >> options ACCEPT_FILTER_HTTP
> >>
> >> options ALTQ
> >>
> >> options ALTQ_CBQ
> >> options ALTQ_RED
> >> options ALTQ_RIO
> >> options ALTQ_HFSC
> >> options ALTQ_PRIQ
> >>
> >> device crypto
> >> device enc
> >>
> >>
> >>> * pciconf -lvc (only include the em(4) items please)
> >> e...@pci0:2:0:0:   class=0x02 card=0x060a15d9 chip=0x10d38086 rev=0x00
> >> hdr=0x00
> >> vendor = 'Intel Corporation'
> >> device = 'Intel 82574L Gigabit Ethernet Controller (82574L)'
> >> class  = network
> >> subclass   = ethernet
> >> cap 01[c8] = powerspec 2  supports D0 D3  current D0
> >> cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
> >> cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1)
> >> cap 11[a0] = MSI-X supports 5 messages in map 0x1c
> >> e...@pci0:3:0:0:   class=0x02 card=0x060a15d9 chip=0x10d38086 rev=0x00
> >> hdr=0x00
> >> vendor = 'Intel Corporation'
> >> device = 'Intel 82574L Gigabit Ethernet Controller (82574L)'
> >> class  = network
> >> subclass   = ethernet
> >> cap 01[c8] = powerspec 2  supports D0 D3  current D0
> >> cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
> >> cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1)
> >> cap 11[a0] = MSI-X supports 5 messages in map 0x1c
> >>
> >>> * vmstat -i
> >> interrupt  total   rate
> >> irq1: atkbd0   9  0
> >> cpu0: timer 36544552   1994
> >> irq256: em0 3801  0
> >> irq257: em1 32963909   1799
> >> irq258: ahci0 175662  9
> >> cpu1: timer 36543525   1994
> >> cpu2: timer 36543525   1994
> >> cpu3: timer 36543525   1994
> >> Total  179318508   9786
> >>
> >> There is an shared IPMI interface on em0, but the interface is not used
> >> by FreeBSD. em1 is used by four VLANs. Polling is only in the
> >> Kernelconfig, not activated on the devices.
> > 
> > So much complexity here.  Tracking this down might be difficult.
> > 
> > One thing that does concern me is the interrupt rate for em1.  Jack et
> > al, is this normal?  I don't see this behaviour on my 8.x systems with
> > em(4) driver 7.0.5, but my systems all use 82573E and 82573L, and don't
> > have MSI-X support.
> 
> This is with the current settings, RXCSUM,TXCSUM and TSO disabled.
> Uptime is now only 5h:44m. It crashes about 2 to 3 times a day. I
> haven't found a way to trigger this, so I can only wait for it happen again.
> 

I'm not sure but it does not look like em(4) issue. Of course em(4)
could free/poke mbuf which was already passed to upper stack but
this may happen only when it has severe bug and I didn't see such
code yet.
It seems you're using ipsec, is it doable for you to disable ipsec
to narrow down the issue?
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Crashes on X7SPE-HF with em

[Philipp Wuensche]

Jeremy Chadwick wrote:
> On Thu, Aug 26, 2010 at 11:56:48PM +0200, Philipp Wuensche wrote:
>> Jeremy Chadwick wrote:
>>> CC'ing Jack Vogel of Intel and Yong-Hyeon PYUN who might have some
>>> ideas.  OP's backtrace is here:
>>>
>>> http://lists.freebsd.org/pipermail/freebsd-stable/2010-August/058425.html
>>>
>>> Philipp, can you please provide the following output?
>>>
>>> * dmesg | egrep 'em[0-9]'
>> em0:  port 0xdc00-0xdc1f mem
>> 0xfe9e-0xfe9f,0xfe9dc000-0xfe9d irq 16 at device 0.0 on pci2
>> em0: Using MSI interrupt
>> em0: [FILTER]
>> em0: Ethernet address: 00:25:90:04:6e:fa
>> em1:  port 0xec00-0xec1f mem
>> 0xfeae-0xfeaf,0xfeadc000-0xfead irq 17 at device 0.0 on pci3
>> em1: Using MSI interrupt
>> em1: [FILTER]
>> em1: Ethernet address: 00:25:90:04:6e:fb
>>
>>> * uname -a (you can XXX out the machine name if need be)
>> FreeBSD XXX 8.1-STABLE FreeBSD 8.1-STABLE #2: Wed Aug 25 10:38:50 CEST
>> 2010 r...@xxx:/usr/obj/usr/src/sys/XXX  amd64
>>
>> Date of source is Aug 17 14:09 CEST 2010. It happend with 8.1-RELEASE
>> too, I can go back to RELEASE or any SVN revision you would like, if it
>> is helping in any way.
>>
>> Kernel-config:
>>
>> include GENERIC
>>
>> ident   XXX
>>
>> options IPSEC
>>
>> options  DEVICE_POLLING
>> options ACCEPT_FILTER_HTTP
>>
>> options ALTQ
>>
>> options ALTQ_CBQ
>> options ALTQ_RED
>> options ALTQ_RIO
>> options ALTQ_HFSC
>> options ALTQ_PRIQ
>>
>> device   crypto
>> device   enc
>>
>>
>>> * pciconf -lvc (only include the em(4) items please)
>> e...@pci0:2:0:0: class=0x02 card=0x060a15d9 chip=0x10d38086 rev=0x00
>> hdr=0x00
>> vendor = 'Intel Corporation'
>> device = 'Intel 82574L Gigabit Ethernet Controller (82574L)'
>> class  = network
>> subclass   = ethernet
>> cap 01[c8] = powerspec 2  supports D0 D3  current D0
>> cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
>> cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1)
>> cap 11[a0] = MSI-X supports 5 messages in map 0x1c
>> e...@pci0:3:0:0: class=0x02 card=0x060a15d9 chip=0x10d38086 rev=0x00
>> hdr=0x00
>> vendor = 'Intel Corporation'
>> device = 'Intel 82574L Gigabit Ethernet Controller (82574L)'
>> class  = network
>> subclass   = ethernet
>> cap 01[c8] = powerspec 2  supports D0 D3  current D0
>> cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
>> cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1)
>> cap 11[a0] = MSI-X supports 5 messages in map 0x1c
>>
>>> * vmstat -i
>> interrupt  total   rate
>> irq1: atkbd0   9  0
>> cpu0: timer 36544552   1994
>> irq256: em0 3801  0
>> irq257: em1 32963909   1799
>> irq258: ahci0 175662  9
>> cpu1: timer 36543525   1994
>> cpu2: timer 36543525   1994
>> cpu3: timer 36543525   1994
>> Total  179318508   9786
>>
>> There is an shared IPMI interface on em0, but the interface is not used
>> by FreeBSD. em1 is used by four VLANs. Polling is only in the
>> Kernelconfig, not activated on the devices.
> 
> So much complexity here.  Tracking this down might be difficult.
> 
> One thing that does concern me is the interrupt rate for em1.  Jack et
> al, is this normal?  I don't see this behaviour on my 8.x systems with
> em(4) driver 7.0.5, but my systems all use 82573E and 82573L, and don't
> have MSI-X support.

This is with the current settings, RXCSUM,TXCSUM and TSO disabled.
Uptime is now only 5h:44m. It crashes about 2 to 3 times a day. I
haven't found a way to trigger this, so I can only wait for it happen again.

Greetings,
philipp
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Crashes on X7SPE-HF with em

[Jeremy Chadwick]

On Thu, Aug 26, 2010 at 11:56:48PM +0200, Philipp Wuensche wrote:
> Jeremy Chadwick wrote:
> > 
> > CC'ing Jack Vogel of Intel and Yong-Hyeon PYUN who might have some
> > ideas.  OP's backtrace is here:
> > 
> > http://lists.freebsd.org/pipermail/freebsd-stable/2010-August/058425.html
> > 
> > Philipp, can you please provide the following output?
> > 
> > * dmesg | egrep 'em[0-9]'
> 
> em0:  port 0xdc00-0xdc1f mem
> 0xfe9e-0xfe9f,0xfe9dc000-0xfe9d irq 16 at device 0.0 on pci2
> em0: Using MSI interrupt
> em0: [FILTER]
> em0: Ethernet address: 00:25:90:04:6e:fa
> em1:  port 0xec00-0xec1f mem
> 0xfeae-0xfeaf,0xfeadc000-0xfead irq 17 at device 0.0 on pci3
> em1: Using MSI interrupt
> em1: [FILTER]
> em1: Ethernet address: 00:25:90:04:6e:fb
> 
> > * uname -a (you can XXX out the machine name if need be)
> 
> FreeBSD XXX 8.1-STABLE FreeBSD 8.1-STABLE #2: Wed Aug 25 10:38:50 CEST
> 2010 r...@xxx:/usr/obj/usr/src/sys/XXX  amd64
> 
> Date of source is Aug 17 14:09 CEST 2010. It happend with 8.1-RELEASE
> too, I can go back to RELEASE or any SVN revision you would like, if it
> is helping in any way.
> 
> Kernel-config:
> 
> include GENERIC
> 
> ident   XXX
> 
> options IPSEC
> 
> options   DEVICE_POLLING
> options ACCEPT_FILTER_HTTP
> 
> options ALTQ
> 
> options ALTQ_CBQ
> options ALTQ_RED
> options ALTQ_RIO
> options ALTQ_HFSC
> options ALTQ_PRIQ
> 
> devicecrypto
> deviceenc
> 
> 
> > * pciconf -lvc (only include the em(4) items please)
> 
> e...@pci0:2:0:0:  class=0x02 card=0x060a15d9 chip=0x10d38086 rev=0x00
> hdr=0x00
> vendor = 'Intel Corporation'
> device = 'Intel 82574L Gigabit Ethernet Controller (82574L)'
> class  = network
> subclass   = ethernet
> cap 01[c8] = powerspec 2  supports D0 D3  current D0
> cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
> cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1)
> cap 11[a0] = MSI-X supports 5 messages in map 0x1c
> e...@pci0:3:0:0:  class=0x02 card=0x060a15d9 chip=0x10d38086 rev=0x00
> hdr=0x00
> vendor = 'Intel Corporation'
> device = 'Intel 82574L Gigabit Ethernet Controller (82574L)'
> class  = network
> subclass   = ethernet
> cap 01[c8] = powerspec 2  supports D0 D3  current D0
> cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
> cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1)
> cap 11[a0] = MSI-X supports 5 messages in map 0x1c
> 
> > * vmstat -i
> 
> interrupt  total   rate
> irq1: atkbd0   9  0
> cpu0: timer 36544552   1994
> irq256: em0 3801  0
> irq257: em1 32963909   1799
> irq258: ahci0 175662  9
> cpu1: timer 36543525   1994
> cpu2: timer 36543525   1994
> cpu3: timer 36543525   1994
> Total  179318508   9786
> 
> There is an shared IPMI interface on em0, but the interface is not used
> by FreeBSD. em1 is used by four VLANs. Polling is only in the
> Kernelconfig, not activated on the devices.

So much complexity here.  Tracking this down might be difficult.

One thing that does concern me is the interrupt rate for em1.  Jack et
al, is this normal?  I don't see this behaviour on my 8.x systems with
em(4) driver 7.0.5, but my systems all use 82573E and 82573L, and don't
have MSI-X support.

-- 
| Jeremy Chadwick   j...@parodius.com |
| Parodius Networking   http://www.parodius.com/ |
| UNIX Systems Administrator  Mountain View, CA, USA |
| Making life hard for others since 1977.  PGP: 4BD6C0CB |

___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Crashes on X7SPE-HF with em

[Philipp Wuensche]

Jack Vogel wrote:
> Hmmm, can you remove ALTQ from the mix and see if that eliminates it?

Sure, but its not used in the pf-rules anyway.

Greetings,
philipp
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Crashes on X7SPE-HF with em

[Jack Vogel]

Hmmm, can you remove ALTQ from the mix and see if that eliminates it?

Jack


On Thu, Aug 26, 2010 at 2:56 PM, Philipp Wuensche wrote:

> Jeremy Chadwick wrote:
> >
> > CC'ing Jack Vogel of Intel and Yong-Hyeon PYUN who might have some
> > ideas.  OP's backtrace is here:
> >
> >
> http://lists.freebsd.org/pipermail/freebsd-stable/2010-August/058425.html
> >
> > Philipp, can you please provide the following output?
> >
> > * dmesg | egrep 'em[0-9]'
>
> em0:  port 0xdc00-0xdc1f mem
> 0xfe9e-0xfe9f,0xfe9dc000-0xfe9d irq 16 at device 0.0 on pci2
> em0: Using MSI interrupt
> em0: [FILTER]
> em0: Ethernet address: 00:25:90:04:6e:fa
> em1:  port 0xec00-0xec1f mem
> 0xfeae-0xfeaf,0xfeadc000-0xfead irq 17 at device 0.0 on pci3
> em1: Using MSI interrupt
> em1: [FILTER]
> em1: Ethernet address: 00:25:90:04:6e:fb
>
> > * uname -a (you can XXX out the machine name if need be)
>
> FreeBSD XXX 8.1-STABLE FreeBSD 8.1-STABLE #2: Wed Aug 25 10:38:50 CEST
> 2010 r...@xxx:/usr/obj/usr/src/sys/XXX  amd64
>
> Date of source is Aug 17 14:09 CEST 2010. It happend with 8.1-RELEASE
> too, I can go back to RELEASE or any SVN revision you would like, if it
> is helping in any way.
>
> Kernel-config:
>
> include GENERIC
>
> ident   XXX
>
> options IPSEC
>
> options DEVICE_POLLING
> options ACCEPT_FILTER_HTTP
>
> options ALTQ
>
> options ALTQ_CBQ
> options ALTQ_RED
> options ALTQ_RIO
> options ALTQ_HFSC
> options ALTQ_PRIQ
>
> device  crypto
> device  enc
>
>
> > * pciconf -lvc (only include the em(4) items please)
>
> e...@pci0:2:0:0: class=0x02 card=0x060a15d9 chip=0x10d38086 rev=0x00
> hdr=0x00
>vendor = 'Intel Corporation'
>device = 'Intel 82574L Gigabit Ethernet Controller (82574L)'
>class  = network
>subclass   = ethernet
>cap 01[c8] = powerspec 2  supports D0 D3  current D0
>cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
>cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1)
>cap 11[a0] = MSI-X supports 5 messages in map 0x1c
> e...@pci0:3:0:0: class=0x02 card=0x060a15d9 chip=0x10d38086 rev=0x00
> hdr=0x00
>vendor = 'Intel Corporation'
>device = 'Intel 82574L Gigabit Ethernet Controller (82574L)'
>class  = network
>subclass   = ethernet
>cap 01[c8] = powerspec 2  supports D0 D3  current D0
>cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
>cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1)
>cap 11[a0] = MSI-X supports 5 messages in map 0x1c
>
> > * vmstat -i
>
> interrupt  total   rate
> irq1: atkbd0   9  0
> cpu0: timer 36544552   1994
> irq256: em0 3801  0
> irq257: em1 32963909   1799
> irq258: ahci0 175662  9
> cpu1: timer 36543525   1994
> cpu2: timer 36543525   1994
> cpu3: timer 36543525   1994
> Total  179318508   9786
>
> There is an shared IPMI interface on em0, but the interface is not used
> by FreeBSD. em1 is used by four VLANs. Polling is only in the
> Kernelconfig, not activated on the devices.
>
> Greetings,
> philipp
>
>
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Crashes on X7SPE-HF with em

[Philipp Wuensche]

Jeremy Chadwick wrote:
> 
> CC'ing Jack Vogel of Intel and Yong-Hyeon PYUN who might have some
> ideas.  OP's backtrace is here:
> 
> http://lists.freebsd.org/pipermail/freebsd-stable/2010-August/058425.html
> 
> Philipp, can you please provide the following output?
> 
> * dmesg | egrep 'em[0-9]'

em0:  port 0xdc00-0xdc1f mem
0xfe9e-0xfe9f,0xfe9dc000-0xfe9d irq 16 at device 0.0 on pci2
em0: Using MSI interrupt
em0: [FILTER]
em0: Ethernet address: 00:25:90:04:6e:fa
em1:  port 0xec00-0xec1f mem
0xfeae-0xfeaf,0xfeadc000-0xfead irq 17 at device 0.0 on pci3
em1: Using MSI interrupt
em1: [FILTER]
em1: Ethernet address: 00:25:90:04:6e:fb

> * uname -a (you can XXX out the machine name if need be)

FreeBSD XXX 8.1-STABLE FreeBSD 8.1-STABLE #2: Wed Aug 25 10:38:50 CEST
2010 r...@xxx:/usr/obj/usr/src/sys/XXX  amd64

Date of source is Aug 17 14:09 CEST 2010. It happend with 8.1-RELEASE
too, I can go back to RELEASE or any SVN revision you would like, if it
is helping in any way.

Kernel-config:

include GENERIC

ident   XXX

options IPSEC

options DEVICE_POLLING
options ACCEPT_FILTER_HTTP

options ALTQ

options ALTQ_CBQ
options ALTQ_RED
options ALTQ_RIO
options ALTQ_HFSC
options ALTQ_PRIQ

device  crypto
device  enc


> * pciconf -lvc (only include the em(4) items please)

e...@pci0:2:0:0:class=0x02 card=0x060a15d9 chip=0x10d38086 rev=0x00
hdr=0x00
vendor = 'Intel Corporation'
device = 'Intel 82574L Gigabit Ethernet Controller (82574L)'
class  = network
subclass   = ethernet
cap 01[c8] = powerspec 2  supports D0 D3  current D0
cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1)
cap 11[a0] = MSI-X supports 5 messages in map 0x1c
e...@pci0:3:0:0:class=0x02 card=0x060a15d9 chip=0x10d38086 rev=0x00
hdr=0x00
vendor = 'Intel Corporation'
device = 'Intel 82574L Gigabit Ethernet Controller (82574L)'
class  = network
subclass   = ethernet
cap 01[c8] = powerspec 2  supports D0 D3  current D0
cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message
cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1)
cap 11[a0] = MSI-X supports 5 messages in map 0x1c

> * vmstat -i

interrupt  total   rate
irq1: atkbd0   9  0
cpu0: timer 36544552   1994
irq256: em0 3801  0
irq257: em1 32963909   1799
irq258: ahci0 175662  9
cpu1: timer 36543525   1994
cpu2: timer 36543525   1994
cpu3: timer 36543525   1994
Total  179318508   9786

There is an shared IPMI interface on em0, but the interface is not used
by FreeBSD. em1 is used by four VLANs. Polling is only in the
Kernelconfig, not activated on the devices.

Greetings,
philipp

___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Crashes on X7SPE-HF with em

[Jeremy Chadwick]

On Thu, Aug 26, 2010 at 07:19:34PM +0200, Philipp Wuensche wrote:
> Mike Tancsa wrote:
> > At 06:55 PM 8/24/2010, Philipp Wuensche wrote:
> >> Hi,
> >>
> >> I'm having trouble with a system on a Supermicro X7SPE-HF, it crashes
> >> about once a day. I haven't found a way to trigger this yet.
> >>
> >> The system has a bunch of VLANs on em1, it does routing between them.
> >>
> >> Currently its running 8-STABLE but it happend with 8.1-RELEASE too.
> > 
> > I dont think its the same problem you are seeing, but the patch in
> > http://lists.freebsd.org/pipermail/freebsd-stable/2010-August/058296.html
> > 
> > might be worth a try.
> 
> Didn't help, crashed again but differently.
> 
> I'm getting a lot of "discard frame w/o packet header" on the em devices
> too.
> 
> I now disabled TSO, RX- and TXCSUM on the device to see if the problems
> comes from this direction.

CC'ing Jack Vogel of Intel and Yong-Hyeon PYUN who might have some
ideas.  OP's backtrace is here:

http://lists.freebsd.org/pipermail/freebsd-stable/2010-August/058425.html

Philipp, can you please provide the following output?

* dmesg | egrep 'em[0-9]'
* uname -a (you can XXX out the machine name if need be)
* pciconf -lvc (only include the em(4) items please)
* vmstat -i

Thanks.

-- 
| Jeremy Chadwick   j...@parodius.com |
| Parodius Networking   http://www.parodius.com/ |
| UNIX Systems Administrator  Mountain View, CA, USA |
| Making life hard for others since 1977.  PGP: 4BD6C0CB |

___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Crashes on X7SPE-HF with em

[Philipp Wuensche]

Mike Tancsa wrote:
> At 06:55 PM 8/24/2010, Philipp Wuensche wrote:
>> Hi,
>>
>> I'm having trouble with a system on a Supermicro X7SPE-HF, it crashes
>> about once a day. I haven't found a way to trigger this yet.
>>
>> The system has a bunch of VLANs on em1, it does routing between them.
>>
>> Currently its running 8-STABLE but it happend with 8.1-RELEASE too.
> 
> I dont think its the same problem you are seeing, but the patch in
> http://lists.freebsd.org/pipermail/freebsd-stable/2010-August/058296.html
> 
> might be worth a try.

Didn't help, crashed again but differently.

I'm getting a lot of "discard frame w/o packet header" on the em devices
too.

I now disabled TSO, RX- and TXCSUM on the device to see if the problems
comes from this direction.

Greetings,
philipp
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: Crashes on X7SPE-HF with em

[Mike Tancsa]

At 06:55 PM 8/24/2010, Philipp Wuensche wrote:

Hi,

I'm having trouble with a system on a Supermicro X7SPE-HF, it crashes
about once a day. I haven't found a way to trigger this yet.

The system has a bunch of VLANs on em1, it does routing between them.

Currently its running 8-STABLE but it happend with 8.1-RELEASE too.


I dont think its the same problem you are seeing, but the patch in
http://lists.freebsd.org/pipermail/freebsd-stable/2010-August/058296.html 


might be worth a try.

---Mike




greetings,
Philipp

# kgdb kernel.debug /var/crash/vmcore.0
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:


Fatal trap 9: general protection fault while in kernel mode
cpuid = 0; apic id = 00
instruction pointer = 0x20:0x8061f5a8
stack pointer   = 0x28:0xff8e64d0
frame pointer   = 0x28:0xff8e64e0
code segment= base 0x0, limit 0xf, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags= interrupt enabled, resume, IOPL = 0
current process = 0 (em1 taskq)
trap number = 9
panic: general protection fault
cpuid = 0
Uptime: 13h24m39s
Physical memory: 4079 MB
Dumping 1349 MB: 1334 1318 1302 1286 1270 1254 1238 1222 1206 1190 1174
1158 1142 1126 1110 1094 1078 1062 1046 1030 1014 998 982 966 950 934
918 902 886 870 854 838 822 806 790 774 758 742 726 710 694 678 662 646
630 614 598 582 566 550 534 518 502 486 470 454 438 422 406 390 374 358
342 326 310 294 278 262 246 230 214 198 182 166 150 134 118 102 86 70 54
38 22 6

Reading symbols from /boot/kernel/zfs.ko...Reading symbols from
/boot/kernel/zfs.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from
/boot/kernel/opensolaris.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
Reading symbols from /boot/kernel/coretemp.ko...Reading symbols from
/boot/kernel/coretemp.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/coretemp.ko
Reading symbols from /boot/kernel/ahci.ko...Reading symbols from
/boot/kernel/ahci.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ahci.ko
Reading symbols from /boot/kernel/ipmi.ko...Reading symbols from
/boot/kernel/ipmi.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ipmi.ko
Reading symbols from /boot/kernel/smbus.ko...Reading symbols from
/boot/kernel/smbus.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/smbus.ko
Reading symbols from /boot/kernel/pflog.ko...Reading symbols from
/boot/kernel/pflog.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/pflog.ko
Reading symbols from /boot/kernel/pf.ko...Reading symbols from
/boot/kernel/pf.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/pf.ko
#0  doadump () at pcpu.h:224
224 __asm("movq %%gs:0,%0" : "=r" (td));
(kgdb) list *0x8061f5a8
0x8061f5a8 is in m_tag_locate (/usr/src/sys/kern/uipc_mbuf2.c:389).
384 if (t == NULL)
385 p = SLIST_FIRST(&m->m_pkthdr.tags);
386 else
387 p = SLIST_NEXT(t, m_tag_link);
388 while (p != NULL) {
389 if (p->m_tag_cookie == cookie && p->m_tag_id == type)
390 return p;
391 p = SLIST_NEXT(p, m_tag_link);
392 }
393 return NULL;
(kgdb) backtrace
#0  doadump () at pcpu.h:224
#1  0x805c25ce in boot (howto=260)
at /usr/src/sys/kern/kern_shutdown.c:416
#2  0x805c29dc in panic (fmt=0x0)
at /usr/src/sys/kern/kern_shutdown.c:590
#3  0x808d40bd in trap_fatal (frame=0x80c8af60,
eva=Variable "eva" is not available.
)
at /usr/src/sys/amd64/amd64/trap.c:777
#4  0x808d4a8b in trap (frame=0xff8e6420)
at /usr/src/sys/amd64/amd64/trap.c:588
#5  0x808b9d64 in calltrap ()
at /usr/src/sys/amd64/amd64/exception.S:224
#6  0x8061f5a8 in m_tag_locate (m=0xff010bb45c00, cookie=0,
type=6, t=Variable "t" is not available.
) at /usr/src/sys/kern/uipc_mbuf2.c:388
#7  0x806d7c56 in ip_ipsec_output (m=0xff8e6598,
inp=0xff010be43150, flags=0xff8e6594,
error=0xff8e65a8, ifp=Variable "ifp" is not available.
) at mbuf.h:1006
#8  0x806d97ef in ip_output (m=0xff010bb45c00, opt=Variable
"opt" is not available.
)
at /usr/src/sys/netinet/ip_output.c:483
#9  0x8073ef13 in tcp_output (tp=0xff000a9eb370)
at /usr/src/