Hello all,
I tried to compile my own kernel. i386 kernel did not work for me. I
tried both GENERIC and GENERIC.MP for amd64. After it, I compiled my own
kernel.
If it is interesting for somebody, I am sending my configs and dmesg. My
questions are (Sorry I am not multiprocessor expert):
- is irq routing ok?
- if not, is there some possibility to make it good?
Result after all this is, that all irqs going from network interfaces
are directed to cpu0 which is overloaded but cpu1 is idle. I think this
is not good.
Thank you for any suggestion !
Lukas Macura
Here is GENERIC.MP (GENERIC untouched):
include "arch/amd64/conf/GENERIC"
option MULTIPROCESSOR
option MPBIOS
option MPVERBOSE
option MPACPI
cpu* at mainbus?
ioapic* at mainbus?
aapic* at pci? # AMD 8131 IO apic
Here is dmesg:
dmesg
OpenBSD 3.8-current (GENERIC.MP) #6: Thu Nov 3 17:32:14 CET 2005
[EMAIL PROTECTED]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 1073319936 (1048164K)
avail mem = 908783616 (887484K)
using 22937 buffers containing 107540480 bytes (105020K) of memory
mainbus0 (root)
mainbus0: scanning 0x9d400 to 0x9d7f0 for MP signature
mainbus0: MP floating pointer found in extended bios data area at
0x9d540
mainbus0: MP config table at 0x9e520, 356 bytes long
mainbus0: Intel MP Specification (Version 1.4) (IBM ENSW X336 SMP )
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(TM) CPU 3.20GHz, 3200.72 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,NXE,LONG
cpu0: 2MB 64b/line 8-way L2 cache
cpu0: calibrating local timer
cpu0: apic clock running at 200007132Hz
cpu0: kstack at 0xffff80006585c000 for 20480 bytes
cpu0: idle pcb at 0xffff80006585c000, idle sp at 0xffff800065860ff0
cpu1 at mainbus0: apid 6 (application processor)
cpu1: Intel(R) Xeon(TM) CPU 3.20GHz, 3200.12 MHz
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,NXE,LONG
cpu1: 2MB 64b/line 8-way L2 cache
cpu1: kstack at 0xffff800065861000 for 20480 bytes
cpu1: idle pcb at 0xffff800065861000, idle sp at 0xffff800065865ff0
mpbios: bus 0 is type PCI
mpbios: bus 1 is type PCI
mpbios: bus 2 is type PCI
mpbios: bus 3 is type PCI
mpbios: bus 4 is type PCI
mpbios: bus 5 is type PCI
mpbios: bus 6 is type PCI
mpbios: bus 7 is type PCI
mpbios: bus 8 is type ISA
ioapic0 at mainbus0 apid 14: pa 0xffff800001ba7f24, virtual wire mode,
version 20, 24 pins
ioapic1 at mainbus0 apid 13: pa 0xffff800001ba7e24, virtual wire mode,
version 20, 24 pins
ioapic2 at mainbus0 apid 12: pa 0xffff800001ba7d24, virtual wire mode,
version 20, 24 pins
ioapic0: int1 attached to isa0 irq 1 (type 0x0 flags 0x0)
ioapic0: int2 attached to isa0 irq 0 (type 0x0 flags 0x0)
ioapic0: int6 attached to isa0 irq 6 (type 0x0 flags 0x0)
ioapic0: int8 attached to isa0 irq 8 (type 0x0 flags 0x5)
ioapic0: int9 attached to isa0 irq 9 (type 0x0 flags 0x0)
ioapic0: int12 attached to isa0 irq 12 (type 0x0 flags 0x0)
ioapic0: int13 attached to isa0 irq 13 (type 0x0 flags 0x0)
ioapic0: int14 attached to isa0 irq 14 (type 0x0 flags 0x0)
ioapic0: int15 attached to isa0 irq 15 (type 0x0 flags 0x0)
mpbios: can't find ioapic 0
ioapic0: int16 attached to pci0 device 29 INT_A (type 0x0 flags 0x0)
ioapic0: int19 attached to pci0 device 29 INT_B (type 0x0 flags 0x0)
ioapic0: int23 attached to pci0 device 29 INT_D (type 0x0 flags 0x0)
ioapic0: int17 attached to pci0 device 31 INT_B (type 0x0 flags 0x0)
ioapic0: int16 attached to pci1 device 1 INT_A (type 0x0 flags 0x0)
mpbios: can't find ioapic 0
ioapic1: int4 attached to pci4 device 1 INT_A (type 0x0 flags 0x0)
ioapic2: int0 attached to pci5 device 1 INT_A (type 0x0 flags 0x0)
ioapic0: int16 attached to pci6 device 0 INT_A (type 0x0 flags 0x0)
ioapic0: int16 attached to pci7 device 0 INT_A (type 0x0 flags 0x0)
local apic: int1 attached to NMI (type 0x1 flags 0x0)
local apic: int0 attached to ExtINT (type 0x3 flags 0x0)
mainbus0: MP WARNING: 348 bytes of extended entries not examined
pci0 at mainbus0 bus 0: configuration mode 1
pchb0 at pci0 dev 0 function 0 "Intel E7710 SMCH" rev 0x0c
"Intel E7710 MCH ERR" rev 0x0c at pci0 dev 0 function 1 not configured
ppb0 at pci0 dev 2 function 0 "Intel E7710 MCH PCIE" rev 0x0c
pci1 at ppb0 bus 2
ppb1 at pci0 dev 4 function 0 "Intel E7710 MCH PCIE" rev 0x0c
pci2 at ppb1 bus 3
ppb2 at pci2 dev 0 function 0 "Intel PCIE-PCIE" rev 0x09
pci3 at ppb2 bus 4
mpt0 at pci3 dev 1 function 0 "Symbios Logic 53c1030" rev 0x08: apic 13
int 4 (irq 10)
mpt0: sending FW Upload request to IOC (size: 36, img size: 69956)
mpt0: IM support: 4
scsibus0 at mpt0: 16 targets
sd0 at scsibus0 targ 0 lun 0: <LSILOGIC, 1030 IM IM, 1000> SCSI2
0/direct fixed
sd0: 139898MB, 139898 cyl, 16 head, 128 sec, 512 bytes/sec, 286511104
sec total
mpt0: target 0 Asynchronous at 0MHz width 8bit offset 0 QAS 0 DT 0 IU 0
ppb3 at pci2 dev 0 function 2 "Intel PCIE-PCIE" rev 0x09
pci4 at ppb3 bus 5
em0 at pci4 dev 1 function 0 "Intel PRO/1000MT (82545GM)" rev 0x04: apic
12 int 0 (irq 10), address 00:0e:0c:9c:07:13
ppb4 at pci0 dev 6 function 0 "Intel E7710 MCH PCIE" rev 0x0c
pci5 at ppb4 bus 6
bge0 at pci5 dev 0 function 0 "Broadcom BCM5721" rev 0x11, BCM5750 B1
(0x4101): apic 14 int 16 (irq 10) address 00:14:5e:0b:3e:ea
brgphy0 at bge0 phy 1: BCM5750 10/100/1000baseT PHY, rev. 0
ppb5 at pci0 dev 7 function 0 "Intel E7710 MCH PCIE" rev 0x0c
pci6 at ppb5 bus 7
bge1 at pci6 dev 0 function 0 "Broadcom BCM5721" rev 0x11, BCM5750 B1
(0x4101): apic 14 int 16 (irq 10) address 00:14:5e:0b:3e:eb
brgphy1 at bge1 phy 1: BCM5750 10/100/1000baseT PHY, rev. 0
vendor "Intel", unknown product 0x359b (class system subclass
miscellaneous, rev 0x0c) at pci0 dev 8 function 0 not configured
uhci0 at pci0 dev 29 function 0 "Intel 82801EB/ER USB" rev 0x02: apic 14
int 16 (irq 10)
usb0 at uhci0: USB revision 1.0
uhub0 at usb0
uhub0: Intel UHCI root hub, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1 at pci0 dev 29 function 1 "Intel 82801EB/ER USB" rev 0x02: apic 14
int 19 (irq 7)
usb1 at uhci1: USB revision 1.0
uhub1 at usb1
uhub1: Intel UHCI root hub, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
ehci0 at pci0 dev 29 function 7 "Intel 82801EB/ER USB" rev 0x02: apic 14
int 23 (irq 5)
usb2 at ehci0: USB revision 2.0
uhub2 at usb2
uhub2: Intel EHCI root hub, rev 2.00/1.00, addr 1
uhub2: 4 ports with 4 removable, self powered
ppb6 at pci0 dev 30 function 0 "Intel 82801BA AGP" rev 0xc2
pci7 at ppb6 bus 1
vga1 at pci7 dev 1 function 0 "ATI Radeon VE QY" rev 0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
pcib0 at pci0 dev 31 function 0 "Intel 82801EB/ER LPC" rev 0x02
pciide0 at pci0 dev 31 function 2 "Intel 82801EB SATA" rev 0x02: DMA,
channel 0 configured to compatibility, channel 1 configured to
compatibility
atapiscsi0 at pciide0 channel 0 drive 0
scsibus1 at atapiscsi0: 2 targets
cd0 at scsibus1 targ 0 lun 0: <HL-DT-ST, DVD-ROM GDR8083N, 0L02> SCSI0
5/cdrom removable
cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
"Intel 82801EB/ER SMBus" rev 0x02 at pci0 dev 31 function 3 not
configured
isa0 at pcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pmsi0 at pckbc0 (aux slot)
pckbc0: using irq 12 for aux slot
wsmouse0 at pmsi0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
sysbeep0 at pcppi0
cpu0: prelint0 0x700 0x0
cpu0: prelint1 0x400 0x0
cpu0: timer0 0x300c0 0x0
cpu0: pcint0 0x10000 0x0
cpu0: lint0 0x10700 0x0
cpu0: lint1 0x400 0x0
cpu0: err0 0x10000 0x0
ioapic2: int0 0xa070 0x0
ioapic1: int4 0xa060 0x0
ioapic0: int16 0xa061 0x0
ioapic0: int19 0xa062 0x0
ioapic0: int23 0xa063 0x0
dkcsum: sd0 matches BIOS drive 0x80
root on sd0a
rootdev=0x400 rrootdev=0xd00 rawdev=0xd02
cpu1: prelint0 0x10000 0x0
cpu1: prelint1 0x10000 0x0
cpu1: timer0 0x200c0 0x0
cpu1: pcint0 0x10000 0x0
cpu1: lint0 0x10700 0x0
cpu1: lint1 0x400 0x0
cpu1: err0 0x10000 0x0
On Po, 2005-10-24 at 21:46 +1000, David Gwynne wrote:
> On 23/10/2005, at 7:29 PM, Luka9 Macura wrote:
>
> > Hello all,
> >
> > Thank you for hint, amd64 architecture does work on our HW !
> > Everything was instaled fine (I instaled latest snapshot). But we have
> > another problem.
> >
> > When I look into BIOS, there is no possibility to do good irq routing.
> > BIOS groups almost all devices to irq3 :( I can change irq, but I
> > cannot
> > change number of devices on this interrupt. There is IRQ for both
> > internal ethernet broadcom cards, IRQ for PCIx slot, HDD, ... and
> > more.
>
> Can you try booting the i386 GENERIc kernel with pcibios disabled?
> And like Stuart Henderson suggested, can you also try the GENERIC.MP
> kernel. If that gets stuck at pcibios too, try disabling it on the mp
> kernel as well.
>
> >
> > I think this is very bad. I do not know if it is due to this simple
> > irq
> > but we have this problems:
> >
> > - disk operations are VERY slow (does not matter, no problem for
> > us, it
> > is firewall).
> > - entire traffic is going thru cpu0. Cpu1 is unused. I think that
> > if two
> > cards are on another irq, it could be possible
> > - througnput of entire system with large packets is somewhere at
> > 120mbps. After it, CPU0 spend all time in interrupts.
> > - I think this machine should be firewalling at much more speed.
> >
> > Please can you help me ? Is there any chance how to change irq if BIOS
> > routes it bad? Or is there another way how to use both CPUs for
> > firewalling?
> >
> > Thank you very much !
> >
> > Here is dmesg:
> > OpenBSD 3.8-current (GENERIC.MP) #556: Sat Oct 15 14:22:53 MDT 2005
> > [EMAIL PROTECTED]:/usr/src/sys/arch/amd64/compile/
> > GENERIC.MP
> > real mem = 1073319936 (1048164K)
> > avail mem = 908791808 (887492K)
> > using 22937 buffers containing 107540480 bytes (105020K) of memory
> > mainbus0 (root)
> > mainbus0: Intel MP Specification (Version 1.4) (IBM ENSW X336 SMP )
> > cpu0 at mainbus0: apid 0 (boot processor)
> > cpu0: Intel(R) Xeon(TM) CPU 3.20GHz, 3200.57 MHz
> > cpu0:
> > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE3
> > 6,CFLUS
> > H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,NXE,LONG
> > cpu0: 2MB 64b/line 8-way L2 cache
> > cpu0: apic clock running at 200007107Hz
> > cpu1 at mainbus0: apid 6 (application processor)
> > cpu1: Intel(R) Xeon(TM) CPU 3.20GHz, 3200.12 MHz
> > cpu1:
> > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE3
> > 6,CFLUS
> > H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,NXE,LONG
> > cpu1: 2MB 64b/line 8-way L2 cache
> > mpbios: bus 0 is type PCI
> > mpbios: bus 1 is type PCI
> > mpbios: bus 2 is type PCI
> > mpbios: bus 3 is type PCI
> > mpbios: bus 4 is type PCI
> > mpbios: bus 5 is type PCI
> > mpbios: bus 6 is type PCI
> > mpbios: bus 7 is type PCI
> > mpbios: bus 8 is type ISA
> > ioapic0 at mainbus0 apid 14: pa 0xffff800001ba7f24, version 20, 24
> > pins
> > ioapic1 at mainbus0 apid 13: pa 0xffff800001ba7e24, version 20, 24
> > pins
> > ioapic2 at mainbus0 apid 12: pa 0xffff800001ba7d24, version 20, 24
> > pins
> > mpbios: can't find ioapic 0
> > mpbios: can't find ioapic 0
> > pci0 at mainbus0 bus 0: configuration mode 1
> > pchb0 at pci0 dev 0 function 0 "Intel E7710 SMCH" rev 0x0c
> > "Intel E7710 MCH ERR" rev 0x0c at pci0 dev 0 function 1 not configured
> > ppb0 at pci0 dev 2 function 0 "Intel E7710 MCH PCIE" rev 0x0c
> > pci1 at ppb0 bus 2
> > ppb1 at pci0 dev 4 function 0 "Intel E7710 MCH PCIE" rev 0x0c
> > pci2 at ppb1 bus 3
> > ppb2 at pci2 dev 0 function 0 "Intel PCIE-PCIE" rev 0x09
> > pci3 at ppb2 bus 4
> > mpt0 at pci3 dev 1 function 0 "Symbios Logic 53c1030" rev 0x08:
> > apic 13
> > int 4 (irq 10)
> > mpt0: sending FW Upload request to IOC (size: 36, img size: 69956)
> > mpt0: IM support: 4
> > scsibus0 at mpt0: 16 targets
> > sd0 at scsibus0 targ 0 lun 0: <LSILOGIC, 1030 IM IM, 1000> SCSI2
> > 0/direct fixed
> > sd0: 139898MB, 139898 cyl, 16 head, 128 sec, 512 bytes/sec, 286511104
> > sec total
> > mpt0: target 0 Asynchronous at 0MHz width 8bit offset 0 QAS 0 DT 0
> > IU 0
> > ppb3 at pci2 dev 0 function 2 "Intel PCIE-PCIE" rev 0x09
> > pci4 at ppb3 bus 5
> > em0 at pci4 dev 1 function 0 "Intel PRO/1000MT (82545GM)" rev 0x04:
> > apic
> > 12 int 0 (irq 10), address 00:0e:0c:9c:07:13
> > ppb4 at pci0 dev 6 function 0 "Intel E7710 MCH PCIE" rev 0x0c
> > pci5 at ppb4 bus 6
> > bge0 at pci5 dev 0 function 0 "Broadcom BCM5721" rev 0x11, BCM5750 B1
> > (0x4101): apic 14 int 16 (irq 10) address 00:14:5e:0b:3e:ea
> > brgphy0 at bge0 phy 1: BCM5750 10/100/1000baseT PHY, rev. 0
> > ppb5 at pci0 dev 7 function 0 "Intel E7710 MCH PCIE" rev 0x0c
> > pci6 at ppb5 bus 7
> > bge1 at pci6 dev 0 function 0 "Broadcom BCM5721" rev 0x11, BCM5750 B1
> > (0x4101): apic 14 int 16 (irq 10) address 00:14:5e:0b:3e:eb
> > brgphy1 at bge1 phy 1: BCM5750 10/100/1000baseT PHY, rev. 0
> > vendor "Intel", unknown product 0x359b (class system subclass
> > miscellaneous, rev 0x0c) at pci0 dev 8 function 0 not configured
> > uhci0 at pci0 dev 29 function 0 "Intel 82801EB/ER USB" rev 0x02:
> > apic 14
> > int 16 (irq 10)
> > usb0 at uhci0: USB revision 1.0
> > uhub0 at usb0
> > uhub0: Intel UHCI root hub, rev 1.00/1.00, addr 1
> > uhub0: 2 ports with 2 removable, self powered
> > uhci1 at pci0 dev 29 function 1 "Intel 82801EB/ER USB" rev 0x02:
> > apic 14
> > int 19 (irq 7)
> > usb1 at uhci1: USB revision 1.0
> > uhub1 at usb1
> > uhub1: Intel UHCI root hub, rev 1.00/1.00, addr 1
> > uhub1: 2 ports with 2 removable, self powered
> > ehci0 at pci0 dev 29 function 7 "Intel 82801EB/ER USB" rev 0x02:
> > apic 14
> > int 23 (irq 5)
> > usb2 at ehci0: USB revision 2.0
> > uhub2 at usb2
> > uhub2: Intel EHCI root hub, rev 2.00/1.00, addr 1
> > uhub2: 4 ports with 4 removable, self powered
> > ppb6 at pci0 dev 30 function 0 "Intel 82801BA AGP" rev 0xc2
> > pci7 at ppb6 bus 1
> > vga1 at pci7 dev 1 function 0 "ATI Radeon VE QY" rev 0x00
> > wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
> > wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
> > pcib0 at pci0 dev 31 function 0 "Intel 82801EB/ER LPC" rev 0x02
> > pciide0 at pci0 dev 31 function 2 "Intel 82801EB SATA" rev 0x02: DMA,
> > channel 0 configured to compatibility, channel 1 configured to
> > compatibility
> > atapiscsi0 at pciide0 channel 0 drive 0
> > scsibus1 at atapiscsi0: 2 targets
> > cd0 at scsibus1 targ 0 lun 0: <HL-DT-ST, DVD-ROM GDR8083N, 0L02> SCSI0
> > 5/cdrom removable
> > cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
> > "Intel 82801EB/ER SMBus" rev 0x02 at pci0 dev 31 function 3 not
> > configured
> > isa0 at pcib0
> > isadma0 at isa0
> > pckbc0 at isa0 port 0x60/5
> > pckbd0 at pckbc0 (kbd slot)
> > pckbc0: using irq 1 for kbd slot
> > wskbd0 at pckbd0: console keyboard, using wsdisplay0
> > pmsi0 at pckbc0 (aux slot)
> > pckbc0: using irq 12 for aux slot
> > wsmouse0 at pmsi0 mux 0
> > pcppi0 at isa0 port 0x61
> > spkr0 at pcppi0
> > sysbeep0 at pcppi0
> > dkcsum: sd0 matches BIOS drive 0x80
> > root on sd0a
> > rootdev=0x400 rrootdev=0xd00 rawdev=0xd02
> >
> >
> > On Ne, 2005-10-16 at 19:21 +0200, Reyk Floeter wrote:
> >
> >> hi,
> >>
> >> On Sun, Oct 16, 2005 at 10:15:24AM +0200, Luka? Macura wrote:
> >>
> >>> We just bought new HW for our firewall. It should operate at high
> >>> speeds
> >>> (up to 1Gbps) so we need strong hardware. We got one but we are not
> >>> possible to instal due to some problems in irq routing process.
> >>> Scenario
> >>> is same with 3.7 and 3.8 kernel. Server hangs just after last
> >>> line in
> >>> this screenshot:
> >>> http://uit.opf.slu.cz/Members/macura/Screenshot1.png
> >>> Last text line is:
> >>> pcibios0: PCI Interrupt Router at 0000:31:0 ("Intel 82801EB/ER
> >>> LPC" rev
> >>> 0x00)
> >>>
> >>>
> >>
> >> i'm curious, could you submit as a bug report again (man sendbug(1))?
> >> so we could pick it up and handle it internally. i'll get my hands on
> >> an 346, which is probably similar.
> >>
> >> your machine should support the EM64T extension, did you try to boot
> >> the amd64 port?
> >>
> >> reyk
> >>
> >
> > [demime 1.01d removed an attachment of type application/x-pkcs7-
> > signature which had a name of smime.p7s]
[demime 1.01d removed an attachment of type application/x-pkcs7-signature which
had a name of smime.p7s]
