Re: kernel/6043: panic: rtfree 2
Hi Claudio and Misc@, On Wed, 25 Feb 2009 03:12:09 +0700, Claudio Jeker wrote: On Tue, Feb 24, 2009 at 08:37:22AM +0700, Insan Praja SW wrote: On Tue, 24 Feb 2009 06:20:05 +0700, Claudio Jeker wrote: Could you please run with the following diff and report if you still hit the panics. It seems some drivers (maybe even em) fail to call the link state update function without the needed splsoftnet protection. Using splnet in rt_if_track may be to protective but we can reconsider that if the panic is gone. thanks Hi Claudio, I will test this diff today and I will get back to you with result. Thanks, I got other negative feedback (in other words the panic still happens). So as we're right before release the best thing is to turn the featrue of for now. So that we don't ship broken code in 4.5. Could you test this diff instead? Yeah I did.. and it works.. So it disabled rt_if_track? how 'bout "sending half of the data to nirvana" things on a multipath routing situation? is it affected? Thanks. -- insandotpraja(at)gmaildotcom
Re: kernel/6043: panic: rtfree 2
On Wed, 25 Feb 2009 03:12:09 +0700, Claudio Jeker wrote: On Tue, Feb 24, 2009 at 08:37:22AM +0700, Insan Praja SW wrote: On Tue, 24 Feb 2009 06:20:05 +0700, Claudio Jeker wrote: Could you please run with the following diff and report if you still hit the panics. It seems some drivers (maybe even em) fail to call the link state update function without the needed splsoftnet protection. Using splnet in rt_if_track may be to protective but we can reconsider that if the panic is gone. thanks Hi Claudio, I will test this diff today and I will get back to you with result. Thanks, I got other negative feedback (in other words the panic still happens). Well, same here. panic: rtfree 2 Stopped atDebugger+0x4: leave ddb> trace Debugger (d2ee1f58,d2ab7100,4,e087fc0c,d2ee1f00) at Debugger+0x4 panic (d06f6a96,d2ee1f78,d2ab7100,0,0) at panic+0x55 rtfree (e087fc0c,30,0,d2ab7100) at rtfree+0xbd route_output (d899d100,d8871c70,0,de84cd50,d0202de9) at route_output+0x341 raw_usrreq (d8871c70,9,d899d100,0,0) at raw_usrreq+0xf2 route_usrreq (d8871c70,9,d899d100,0,0) at route_usrreq+0xe8 sosend (d8871c70,0,de84cea8,d899d100,0,0) at sosend+0x357 soo_write (d8a00ec8,d8a00ee4,de84cea8,d8a1e000) at soo_write+0x1c dofilewritev (d89f8c44,6,d8a00ec8,cfbd44905) at dofilewritev+0xeb sys_writev (d89f8c44,de84cf68,de84cf58,cfbd4490,d89f8c44) at sys_writev+0x47 syscall () at syscall+0x24e --- syscall (number 121) --- 0x3623d9d So as we're right before release the best thing is to turn the featrue of for now. So that we don't ship broken code in 4.5. Could you test this diff instead? Ok. Thanks, Insan -- insandotpraja(at)gmaildotcom
Re: panic: rtfree 2
On Wed, 14 Jan 2009 06:11:56 +0700, Claudio Jeker wrote: On Tue, Jan 13, 2009 at 01:43:09PM +0700, Insan Praja wrote: Hi Misc@, On Tue, Jan 13, 2009 at 6:32 AM, Claudio Jeker wrote: > On Tue, Jan 13, 2009 at 05:49:00AM +0700, Insan Praja SW wrote: >> Hi Misc@, >> So I tried to move the cable (and configs) to other interfaces. >> Unplugging the cable will reproduce the problem. So IMHO it must be >> something in bgpd or something in kernel that manages routes. >> Thanks, >> > > Could you please send me the output of netstat -Arn just before you > produce the panic. Hopefully this gives me the hint what goes wrong. > It's a lot of output (full routes), but I could cut some of the first output I would need the full output but I just saw your PR and there you mention multiple routing tables. Did you try without multiple tables? Is bgp writing to an alternate table? I see if I can reproduce it. Bgpd use rtable 0. I use multiple routing table to produce VRF/VPN-like network on my routers. I got rtable 1 and rtable 2. I haven't tested this with single table. A full "netstat -Arn" output would be a 62.5Mb files, a don't think it would pass the mailer. Here's a link http://www.4shared.com/file/80547753/5bb42570/netstat.html for downloading link. Thanks, Insan -- insandotpraja(at)gmaildotcom
Re: panic: rtfree 2
On Tue, Jan 13, 2009 at 01:43:09PM +0700, Insan Praja wrote: > Hi Misc@, > > On Tue, Jan 13, 2009 at 6:32 AM, Claudio Jeker > wrote: > > On Tue, Jan 13, 2009 at 05:49:00AM +0700, Insan Praja SW wrote: > >> Hi Misc@, > >> So I tried to move the cable (and configs) to other interfaces. > >> Unplugging the cable will reproduce the problem. So IMHO it must be > >> something in bgpd or something in kernel that manages routes. > >> Thanks, > >> > > > > Could you please send me the output of netstat -Arn just before you > > produce the panic. Hopefully this gives me the hint what goes wrong. > > > > It's a lot of output (full routes), but I could cut some of the first output > I would need the full output but I just saw your PR and there you mention multiple routing tables. Did you try without multiple tables? Is bgp writing to an alternate table? I see if I can reproduce it. -- :wq Claudio
Re: panic: rtfree 2
Hi Misc@,
On Tue, Jan 13, 2009 at 6:32 AM, Claudio Jeker wrote:
> On Tue, Jan 13, 2009 at 05:49:00AM +0700, Insan Praja SW wrote:
>> Hi Misc@,
>> So I tried to move the cable (and configs) to other interfaces.
>> Unplugging the cable will reproduce the problem. So IMHO it must be
>> something in bgpd or something in kernel that manages routes.
>> Thanks,
>>
>
> Could you please send me the output of netstat -Arn just before you
> produce the panic. Hopefully this gives me the hint what goes wrong.
>
It's a lot of output (full routes), but I could cut some of the first output
Routing tables
Netmasks:
0xd2ab54a0 (root node)
0xd2e1db00 (255) Qff.ff.f0
0xd2e1da00 (255) Qff.ff.ff
0xd2e2b100 (255) Qff.ff.ff.80
0xd2e2bb80 (255) Qff.ff.ff.c0
0xd2e28d80 (255) Qff.ff.ff.e0
0xd2e26a80 (255) Qff.ff.ff.f0
0xd2e25600 (255) Qff.ff.ff.f8
0xd2e25580 (255) Qff.ff.ff.fc
0xd2e26e80 (255) Qff.ff.ff.fe
0xd2e26b00 (255) Qff.ff.ff.ff
0xd2e1d280 (255) Qff.ff.ff.ff.80
0xd2e2b800 (255) Qff.ff.ff.ff.c0
0xd2e2b700 (255) Qff.ff.ff.ff.e0
0xd2e2b200 (255) Qff.ff.ff.ff.f0
0xd2e2be00 (255) Qff.ff.ff.ff.f8
0xd2e2bd80 (255) Qff.ff.ff.ff.fc
0xd2e1df00 (255) Qff.ff.ff.ff.fe
0xd2e1df80 (255) Qff.ff.ff.ff.ff
0xd2e2bd00 (255) Qff.ff.ff.ff.ff.e0
0xd2e1dc00 (255) Qff.ff.ff.ff.ff.f0
0xd2e1d900 (255) Qff.ff.ff.ff.ff.f8
0xd2b75a00 (255) Qff.ff.ff.ff.ff.fc
0xd2b75c80 (255) Qff.ff.ff.ff.ff.ff.ff.c0
0xd2b75480 (255) Qff.ff.ff.ff.ff.ff.ff.ff
0xd2b75780 (255) Qff.ff.ff.ff.ff.ff.ff.ff.f0
0xd2b75680 (255) Qff.ff.ff.ff.ff.ff.ff.ff.ff
0xd2b75080 (255) Qff.ff.ff.ff.ff.ff.ff.ff.ff.ff
0xd2ae5e00 (255) Qff.ff.ff.ff.ff.ff.ff.ff.ff.ff.ff.ff.ff.ff
0xd2b75b80 (255) Qff.ff.ff.ff.ff.ff.ff.ff.ff.ff.ff.ff.ff.ff.ff.ff.ff.ff
0xd2b75a80 (255) Qff.ff.ff.ff.ff.ff.ff.ff.ff.ff.ff.ff.ff.ff.ff.ff.ff.ff.f0
0xd2b75980 (255) Qff.ff.ff.ff.ff.ff.ff.ff.ff.ff.ff.ff.ff.ff.ff.ff.ff.ff.ff
0xd2ab54d0 (root node)
Internet:
AddressDestinationGatewayFlags Refs Use
Mtu Prio Iface
0xd2ab5338 (32) (0xd2ab5338) 0xd98bd018 :
0xd97172ac mk = 0xd2e22be0 {(0), , default}
0xd98bd018 (33) (0xd2ab5338) 0xd976d10c : 0xd97669b8
0xd976d10c (34) (0xd98bd018) 0xd96fe5c4 : 0xd9768188
0xd96fe5c4 (35) (0xd976d10c) 0xd98b3a14 : 0xd956a030
0xd98b3a14 (36) (0xd96fe5c4) 0xd95c0af8 : 0xd976ca94
0xd95c0af8 (37) (0xd98b3a14) 0xe05af698 : 0xe0dbc630
If you need more, I'm willing to send a private message to your email
with the output attached.
> thanks.
>
> --
> :wq Claudio
>
Thanks,
Insan
>
>>
>> Insan
>>
>>> Hi Misc@,
>>> recently installed a i386-current from january 11th 2009, on an intel
>>> s3210SH serverboard. I run pf, bgpd, snmpd and ntpd. When I accidentally
>>> unplugged em0, somehow it went panic and said "panic: rtfree 2". I
>>> entered
>>> ddb and run a trace with the following result:
>>
>>> ddb> trace
>>> Debugger(d39d1c58,d2ab5300,4,e1bc0cfc,d39d1c00) at Debbugger+0x4
>>> panic(d06f5e16,d39d1c78,d2ab5300,0,0) at panic+0x55
>>> rtfree(e1bc0cfc,30,0,d2ab5300) at rtfree+0xbd
>>> route_output(d9898000,d9783c70,0,df75fd50,d0202dd5) at route_output+0x341
>>> raw_usrreq(d9783c70,9,d9898000,0,0) at raw_usrreq+0xf2
>>> route_usrreq(d9783c70,9,d9898000,0,0) at route_usrreq+0xe8
>>> sosend(d9783c70,0,df75fea8,d9898000,0) at sosend+0x357
>>> soo_write(d977f424,d977f440,df75fea8,d992c000) at soo_write+0x1c
>>> dofilewritev(d990ada0,6,d997f424,cfbecfc0,5) at dofilewritev+0xeb
>>> sys_writev(d990ada0,df75ff68,df75ff58,cfbecfc0,d990ada0) at
>>> sys_writev+0x47
>>> syscall() at syscall+0x24e
>>> --- syscall (number 121) ---
>>> 0x74ba3d5
>>> ddb>
>>>
>>> I did run ps and notice some particular things (I'm too exhausted to
>>> retype everything from the picture):
>>> PID PPIDPGRPUID S FLAGS WAITCOMMAND
>>> 6790 6286628675 3 0x180 pollbgpd
>>> 7389 6286628675 2 0x100 bgpd
>>> *62861 62860 7 0 bgpd
>>>
>>> Of course, it's not happen if bgpd is not running and I consciusly
>>> ifconfig-ed em0 to down. But, if I keep bgpd running and ifconfig-ed em0
>>> to down, nothing happen until I ifconfig-ed em0 to up, it went panic with
>>> the same messages. I hope this could helps all of us. Thanks a lot for
>>> everything, I hope I could help more. I also sendbug-ed the messages.
>>> Below is the dmesg.
Re: panic: rtfree 2
Hi Misc@,
So I tried to move the cable (and configs) to other interfaces. Unplugging
the cable will reproduce the problem. So IMHO it must be something in bgpd
or something in kernel that manages routes.
Thanks,
Insan
Hi Misc@,
recently installed a i386-current from january 11th 2009, on an intel
s3210SH serverboard. I run pf, bgpd, snmpd and ntpd. When I accidentally
unplugged em0, somehow it went panic and said "panic: rtfree 2". I
entered
ddb and run a trace with the following result:
ddb> trace
Debugger(d39d1c58,d2ab5300,4,e1bc0cfc,d39d1c00) at Debbugger+0x4
panic(d06f5e16,d39d1c78,d2ab5300,0,0) at panic+0x55
rtfree(e1bc0cfc,30,0,d2ab5300) at rtfree+0xbd
route_output(d9898000,d9783c70,0,df75fd50,d0202dd5) at route_output+0x341
raw_usrreq(d9783c70,9,d9898000,0,0) at raw_usrreq+0xf2
route_usrreq(d9783c70,9,d9898000,0,0) at route_usrreq+0xe8
sosend(d9783c70,0,df75fea8,d9898000,0) at sosend+0x357
soo_write(d977f424,d977f440,df75fea8,d992c000) at soo_write+0x1c
dofilewritev(d990ada0,6,d997f424,cfbecfc0,5) at dofilewritev+0xeb
sys_writev(d990ada0,df75ff68,df75ff58,cfbecfc0,d990ada0) at
sys_writev+0x47
syscall() at syscall+0x24e
--- syscall (number 121) ---
0x74ba3d5
ddb>
I did run ps and notice some particular things (I'm too exhausted to
retype everything from the picture):
PID PPIDPGRPUID S FLAGS WAITCOMMAND
67906286628675 3 0x180 pollbgpd
73896286628675 2 0x100 bgpd
*6286 1 62860 7 0 bgpd
Of course, it's not happen if bgpd is not running and I consciusly
ifconfig-ed em0 to down. But, if I keep bgpd running and ifconfig-ed em0
to down, nothing happen until I ifconfig-ed em0 to up, it went panic with
the same messages. I hope this could helps all of us. Thanks a lot for
everything, I hope I could help more. I also sendbug-ed the messages.
Below is the dmesg.
OpenBSD 4.4-current (GENERIC) #1: Sun Jan 11 23:23:00 WIT 2009
r...@greenbridge.mygreenlinks.net:/usr/src/sys/arch/i386/compile/GENERIC
RTC BIOS diagnostic error d
cpu0: Intel(R) Xeon(R) CPU E3110 @ 3.00GHz ("GenuineIntel" 686-class) 3
GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,S
SE3,MWAIT,DS-CPL,VMX,SMX,EST,TM2,CX16,xTPR
real mem = 2143842304 (2044MB)
avail mem = 2064748544 (1969MB)
RTC BIOS diagnostic error d
mainbus0 at root
bios0 at mainbus0: AT/286+ BIOS, date 12/12/07, SMBIOS rev. 2.5 @
0x7fdfd000 (63 entries)
bios0: vendor Intel Corporation version
"S3200X38.86B.00.00.0045.082820081329" date 08/28/2008
bios0: Intel Corporation S3210SH
acpi0 at bios0: rev 2
acpi0: tables DSDT SLIC FACP APIC WDDT MCFG HPET SPCR SSDT SSDT SSDT SSDT
SSDT HEST BERT ERST EINJ DMAR
acpi0: wakeup devices SLPB(S5) NPE1(S5) NPE6(S5) P32_(S5) PS2M(S1)
PS2K(S1) ILAN(S5) PEX0(S5) PEX1(S5) PEX2(S5) PEX3(S5) PEX4(
S5) PEX5(S5) UHC1(S1) UHC2(S1) UHC3(S1) UHC4(S1) EHCI(S1) EHC2(S1)
UH42(S1) UHC5(S1) UHC6(S1) AZAL(S4)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: apic clock running at 332MHz
cpu at mainbus0: not configured
ioapic0 at mainbus0: apid 5 pa 0xfec0, version 20, 24 pins
ioapic0: misconfigured as apic 0, remapped to apid 5
acpihpet0 at acpi0: 14318179 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus -1 (NPE1)
acpiprt2 at acpi0: bus -1 (NPE6)
acpiprt3 at acpi0: bus 3 (P32_)
acpiprt4 at acpi0: bus 1 (PEX0)
acpiprt5 at acpi0: bus -1 (PEX1)
acpiprt6 at acpi0: bus -1 (PEX2)
acpiprt7 at acpi0: bus -1 (PEX3)
acpiprt8 at acpi0: bus 2 (PEX4)
acpiprt9 at acpi0: bus -1 (PEX5)
acpicpu0 at acpi0: FVS, 3000, 2000 MHz
acpibtn0 at acpi0: SLPB
acpibtn1 at acpi0: PWRB
bios0: ROM list: 0xc/0x8000 0xc8000/0x1000 0xc9000/0x1000
ipmi0 at mainbus0: version 2.0 interface KCS iobase 0xca2/2 spacing 1
pci0 at mainbus0 bus 0: configuration mode 1 (bios)
pchb0 at pci0 dev 0 function 0 "Intel 3200/3210 Host" rev 0x00
em0 at pci0 dev 25 function 0 "Intel ICH9 IGP AMT" rev 0x02: apic 5 int
20
(irq 11), address 00:15:17:28:2a:d3
uhci0 at pci0 dev 26 function 0 "Intel 82801I USB" rev 0x02: apic 5 int
18
(irq 9)
uhci1 at pci0 dev 26 function 1 "Intel 82801I USB" rev 0x02: apic 5 int
21
(irq 10)
ehci0 at pci0 dev 26 function 7 "Intel 82801I USB" rev 0x02: apic 5 int
17
(irq 10)
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb0 at pci0 dev 28 function 0 "Intel 82801I PCIE" rev 0x02: apic 5 int
17
(irq 11)
pci1 at ppb0 bus 1
em1 at pci1 dev 0 function 0 "Intel PRO/1000 PT (82571EB)" rev 0x06: apic
5 int 16 (irq 11), address 00:15:17:86:53:14
em2 at pci1 dev 0 function 1 "Intel PRO/1000 PT (82571EB)" rev 0x06: apic
5 int 17 (irq 10), address 00:15:17:86:53:15
ppb1 at
panic: rtfree 2
Hi Misc@,
recently installed a i386-current from january 11th 2009, on an intel
s3210SH serverboard. I run pf, bgpd, snmpd and ntpd. When I accidentally
unplugged em0, somehow it went panic and said "panic: rtfree 2". I entered
ddb and run a trace with the following result:
ddb> trace
Debugger(d39d1c58,d2ab5300,4,e1bc0cfc,d39d1c00) at Debbugger+0x4
panic(d06f5e16,d39d1c78,d2ab5300,0,0) at panic+0x55
rtfree(e1bc0cfc,30,0,d2ab5300) at rtfree+0xbd
route_output(d9898000,d9783c70,0,df75fd50,d0202dd5) at route_output+0x341
raw_usrreq(d9783c70,9,d9898000,0,0) at raw_usrreq+0xf2
route_usrreq(d9783c70,9,d9898000,0,0) at route_usrreq+0xe8
sosend(d9783c70,0,df75fea8,d9898000,0) at sosend+0x357
soo_write(d977f424,d977f440,df75fea8,d992c000) at soo_write+0x1c
dofilewritev(d990ada0,6,d997f424,cfbecfc0,5) at dofilewritev+0xeb
sys_writev(d990ada0,df75ff68,df75ff58,cfbecfc0,d990ada0) at sys_writev+0x47
syscall() at syscall+0x24e
--- syscall (number 121) ---
0x74ba3d5
ddb>
I did run ps and notice some particular things (I'm too exhausted to
retype everything from the picture):
PID PPIDPGRPUID S FLAGS WAITCOMMAND
67906286628675 3 0x180 pollbgpd
73896286628675 2 0x100 bgpd
*6286 1 62860 7 0 bgpd
Of course, it's not happen if bgpd is not running and I consciusly
ifconfig-ed em0 to down. But, if I keep bgpd running and ifconfig-ed em0
to down, nothing happen until I ifconfig-ed em0 to up, it went panic with
the same messages. I hope this could helps all of us. Thanks a lot for
everything, I hope I could help more. I also sendbug-ed the messages.
Below is the dmesg.
OpenBSD 4.4-current (GENERIC) #1: Sun Jan 11 23:23:00 WIT 2009
r...@greenbridge.mygreenlinks.net:/usr/src/sys/arch/i386/compile/GENERIC
RTC BIOS diagnostic error d
cpu0: Intel(R) Xeon(R) CPU E3110 @ 3.00GHz ("GenuineIntel" 686-class) 3 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,S
SE3,MWAIT,DS-CPL,VMX,SMX,EST,TM2,CX16,xTPR
real mem = 2143842304 (2044MB)
avail mem = 2064748544 (1969MB)
RTC BIOS diagnostic error d
mainbus0 at root
bios0 at mainbus0: AT/286+ BIOS, date 12/12/07, SMBIOS rev. 2.5 @
0x7fdfd000 (63 entries)
bios0: vendor Intel Corporation version
"S3200X38.86B.00.00.0045.082820081329" date 08/28/2008
bios0: Intel Corporation S3210SH
acpi0 at bios0: rev 2
acpi0: tables DSDT SLIC FACP APIC WDDT MCFG HPET SPCR SSDT SSDT SSDT SSDT
SSDT HEST BERT ERST EINJ DMAR
acpi0: wakeup devices SLPB(S5) NPE1(S5) NPE6(S5) P32_(S5) PS2M(S1)
PS2K(S1) ILAN(S5) PEX0(S5) PEX1(S5) PEX2(S5) PEX3(S5) PEX4(
S5) PEX5(S5) UHC1(S1) UHC2(S1) UHC3(S1) UHC4(S1) EHCI(S1) EHC2(S1)
UH42(S1) UHC5(S1) UHC6(S1) AZAL(S4)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee0: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: apic clock running at 332MHz
cpu at mainbus0: not configured
ioapic0 at mainbus0: apid 5 pa 0xfec0, version 20, 24 pins
ioapic0: misconfigured as apic 0, remapped to apid 5
acpihpet0 at acpi0: 14318179 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus -1 (NPE1)
acpiprt2 at acpi0: bus -1 (NPE6)
acpiprt3 at acpi0: bus 3 (P32_)
acpiprt4 at acpi0: bus 1 (PEX0)
acpiprt5 at acpi0: bus -1 (PEX1)
acpiprt6 at acpi0: bus -1 (PEX2)
acpiprt7 at acpi0: bus -1 (PEX3)
acpiprt8 at acpi0: bus 2 (PEX4)
acpiprt9 at acpi0: bus -1 (PEX5)
acpicpu0 at acpi0: FVS, 3000, 2000 MHz
acpibtn0 at acpi0: SLPB
acpibtn1 at acpi0: PWRB
bios0: ROM list: 0xc/0x8000 0xc8000/0x1000 0xc9000/0x1000
ipmi0 at mainbus0: version 2.0 interface KCS iobase 0xca2/2 spacing 1
pci0 at mainbus0 bus 0: configuration mode 1 (bios)
pchb0 at pci0 dev 0 function 0 "Intel 3200/3210 Host" rev 0x00
em0 at pci0 dev 25 function 0 "Intel ICH9 IGP AMT" rev 0x02: apic 5 int 20
(irq 11), address 00:15:17:28:2a:d3
uhci0 at pci0 dev 26 function 0 "Intel 82801I USB" rev 0x02: apic 5 int 18
(irq 9)
uhci1 at pci0 dev 26 function 1 "Intel 82801I USB" rev 0x02: apic 5 int 21
(irq 10)
ehci0 at pci0 dev 26 function 7 "Intel 82801I USB" rev 0x02: apic 5 int 17
(irq 10)
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb0 at pci0 dev 28 function 0 "Intel 82801I PCIE" rev 0x02: apic 5 int 17
(irq 11)
pci1 at ppb0 bus 1
em1 at pci1 dev 0 function 0 "Intel PRO/1000 PT (82571EB)" rev 0x06: apic
5 int 16 (irq 11), address 00:15:17:86:53:14
em2 at pci1 dev 0 function 1 "Intel PRO/1000 PT (82571EB)" rev 0x06: apic
5 int 17 (irq 10), address 00:15:17:86:53:15
ppb1 at pci0 dev 28 function 4 "Intel 82801I PCIE" rev 0x02: apic 5 int 17
(irq 11)
pci2 at ppb1 bus 2
vga1 at pci2 dev 0 function 0 "Matrox MGA G200e (ServerEngines)" rev 0x02
wsdisplay0
