Hi Aakash, First of all, thanks for the many pointers here, I cannot begin to describe the gratitude for your help. My responses to your suggestions below.
On 05/01/2013 08:36 PM, Aakash Saini wrote: >> ..."Strangely, however, prtdiag sees the slots as populated:"... > > I don't understand how it could be pci_boot/create_bus issues; cause pci_boot > is functioning and allocating the buses *correctly* to relevant slots. As > stated, Left-hand has 4::x8-PCI-e slots (2-cpu1, 2-cpu2) and right-hand has > 2::x8-PCI-e slots (2-cpu2), which we see through prtdiag details about > slots (1-to-6) assignments. > > ..which means socket 00 is working correctly, and socket 01 has some > fault!!. surprising, very much nailed correctly relating to QPI > interconnect issues. i guess PCH/C6xxx-chipset and motherboard specs need > to be reviewed. > > prtpicl should display correctly details to resource addressing, cause > slots are assigned. Attached, both the short and verbose version. >> ..."The problem appears to be that Illumos doesn't recognize any PCI > bridges beyond those attached to CPU1. > > yes, maybe its a PCIe-PCI bridging issue, even i doubt. try kdb debug > state... > ::bp pcieb`pcieb_attach > :c > pcieb`pcieb_dbg_print/W 0x1 --- this should display basic > debugged information > :c Compiled a debug version of the pcieb module and attached the output from this. Strangely enough, the output stopped for considerable time between lines 21 and 22 in the output, so between lines: pcieb(0): pwrINITCHILD: config regs setup for pci15d9,690@0 and pcieb(1): pwrpcieb_pwr_disable: disabling PM after which lots of output proceeded quickly. Could it be that the probing code is getting stuck on and then giving up on probing a bridge here? > also. try with disabled (both) MSI and workaround patch to 41210/chipset > individually. > pcieb`pcieb_enable_msi/W 0x1 > pcieb_disable_41210_wkarnd/W 0x1 Neither setting changed anything. > i'm not sure if there is any resource conflict issue; mdb -k | ::interrupts > as obviously, h/w is remote! Here's the output of that: # echo ::interrupts | mdb -k CPU/Vect IRQ IPL Bus Trg Type Share APIC/INT# ISR 1/0x20 9 9 PCI Lvl Fixed 1 0x0/0x9 acpi_wrapper_isr 2/0x20 11 14 PCI Lvl Fixed 1 0x0/0xb hpet_isr 3/0x20 - 7 PCI Edg MSI 1 - pcieb_intr_handler 4/0x20 - 5 PCI Edg MSI-X 1 - mrsas_isr 7/0x20 1 5 ISA Edg Fixed 1 0x0/0x1 i8042_intr 8/0x20 12 5 ISA Edg Fixed 1 0x0/0xc i8042_intr 8/0x22 - 5 PCI Edg MSI 1 - mpt_intr 12/0x20 3 12 ISA Edg Fixed 1 0x0/0x3 asyintr 13/0x20 - 4 PCI Edg MSI 1 - pcieb_intr_handler 14/0x20 - 7 PCI Edg MSI 1 - pcieb_intr_handler 15/0x20 - 4 PCI Edg MSI 1 - pcieb_intr_handler 16/0x20 - 7 PCI Edg MSI 1 - pcieb_intr_handler 17/0x20 - 4 PCI Edg MSI 1 - pcieb_intr_handler 18/0x20 16 9 PCI Lvl Fixed 1 0x0/0x10 ehci_intr 19/0x20 23 9 PCI Lvl Fixed 1 0x0/0x17 ehci_intr 20/0x20 - 6 PCI Edg MSI-X 1 - igb_intr_tx_other 21/0x20 - 6 PCI Edg MSI-X 1 - igb_intr_rx all/0xf0 - 15 - Edg IPI 1 - xc_serv all/0xf1 - 11 - Edg IPI 0 - poke_cpu all/0xf2 - 14 - Edg IPI 1 - kcpc_hw_overflow_intr all/0xf3 - 15 - Edg IPI 1 - apic_error_intr all/0xf4 - 2 - Edg IPI 1 - cmi_cmci_trap all/0xf5 - 14 - Edg IPI 1 - cbe_fire all/0xf6 - 14 - Edg IPI 1 - cbe_fire Hope this provides more insight. If it helps, I can arrange for direct access to the machine (both to the OS and the BMC). Cheers, -- Saso
/ (picl, f900000001)
platform (upa, f900000005)
scsi_vhci (devctl, f90000005c)
pci (pciex, f900000079)
pci15d9,720 (obp-device, f90000008e)
pci8086,3c02 (pciex, f9000000af)
pci15d9,690 (obp-device, f9000000d6)
sd (block, f900000108)
pci8086,3c04 (pciex, f900000126)
pci15d9,10fb (obp-device, f90000014e)
pci15d9,10fb (obp-device, f900000179)
pci8086,3c06 (pciex, f9000001a3)
pci10e3,8113 (pci, f9000001ca)
display (display, f9000001ee)
pci8086,3c07 (pciex, f900000214)
pci15d9,1521 (obp-device, f90000023b)
pci15d9,1521 (obp-device, f900000266)
pci8086,3c08 (pciex, f900000290)
pci103c,322d (obp-device, f9000002b8)
pci8086,3c0a (pciex, f9000002f6)
pci15d9,720 (obp-device, f90000031a)
pci15d9,720 (obp-device, f90000033c)
pci15d9,720 (obp-device, f90000035e)
pci15d9,720 (obp-device, f900000380)
pci15d9,720 (obp-device, f9000003a2)
pci15d9,720 (obp-device, f9000003c4)
pci15d9,720 (obp-device, f9000003e6)
pci15d9,720 (obp-device, f900000408)
pci15d9,720 (obp-device, f90000042a)
pci15d9,720 (obp-device, f90000044a)
pci15d9,720 (obp-device, f90000046a)
pci8086,1d3e (pciex, f900000487)
pci15d9,720 (obp-device, f9000004ad)
pci15d9,720 (obp-device, f9000004cf)
pci15d9,720 (obp-device, f9000004ee)
pci15d9,720 (obp-device, f90000050d)
hub (obp-device, f900000537)
device (obp-device, f900000563)
mouse (obp-device, f900000583)
keyboard (obp-device, f90000059b)
pci15d9,720 (obp-device, f9000005b3)
hub (obp-device, f9000005dd)
storage (obp-device, f900000609)
disk (block, f900000637)
pci8086,244e (pci, f900000657)
isa (isa, f900000679)
motherboard (obp-device, f90000069c)
i8042 (obp-device, f9000006a6)
mouse (obp-device, f9000006b1)
asy (serial, f9000006c0)
asy (serial, f9000006cf)
motherboard (obp-device, f9000006de)
motherboard (obp-device, f9000006e8)
pci15d9,720 (obp-device, f9000006f2)
pci15d9,720 (obp-device, f900000723)
pci15d9,720 (obp-device, f900000742)
pci (pciex, f900000778)
pci15d9,720 (obp-device, f90000078b)
pci15d9,720 (obp-device, f9000007a7)
pci15d9,720 (obp-device, f9000007c7)
pci15d9,720 (obp-device, f9000007e7)
pci15d9,720 (obp-device, f900000803)
pci15d9,720 (obp-device, f900000823)
pci15d9,720 (obp-device, f900000843)
pci15d9,720 (obp-device, f90000085f)
pci15d9,720 (obp-device, f90000087b)
pci15d9,720 (obp-device, f900000897)
pci15d9,720 (obp-device, f9000008b3)
pci15d9,720 (obp-device, f9000008cf)
pci15d9,720 (obp-device, f9000008eb)
pci15d9,720 (obp-device, f900000907)
pci15d9,720 (obp-device, f900000923)
pci15d9,720 (obp-device, f90000093f)
pci15d9,720 (obp-device, f90000095b)
pci15d9,720 (obp-device, f900000977)
pci15d9,720 (obp-device, f900000993)
pci15d9,720 (obp-device, f9000009af)
pci15d9,720 (obp-device, f9000009cb)
pci15d9,720 (obp-device, f9000009e7)
pci15d9,720 (obp-device, f900000a03)
pci15d9,720 (obp-device, f900000a1f)
pci15d9,720 (obp-device, f900000a3f)
pci15d9,720 (obp-device, f900000a5f)
pci15d9,720 (obp-device, f900000a7f)
pci15d9,720 (obp-device, f900000a9f)
pci15d9,720 (obp-device, f900000abf)
pci15d9,720 (obp-device, f900000adf)
pci15d9,720 (obp-device, f900000afb)
pci15d9,720 (obp-device, f900000b1b)
pci15d9,720 (obp-device, f900000b3b)
pci15d9,720 (obp-device, f900000b5b)
pci15d9,720 (obp-device, f900000b7b)
pci15d9,720 (obp-device, f900000b9b)
pci15d9,720 (obp-device, f900000bbb)
pci15d9,720 (obp-device, f900000bdb)
pci8086,0 (obp-device, f900000bfb)
pci15d9,720 (obp-device, f900000c17)
pci15d9,720 (obp-device, f900000c33)
pci15d9,720 (obp-device, f900000c4f)
pci15d9,720 (obp-device, f900000c6b)
pci15d9,720 (obp-device, f900000c87)
pci (pciex, f900000ca3)
pci8086,3c01 (pciex, f900000cb3)
pci8086,3c02 (pciex, f900000cd6)
pci8086,3c04 (pciex, f900000cf9)
pci103c,322d (obp-device, f900000d1d)
pci8086,3c06 (pciex, f900000d40)
pci8086,3c08 (pciex, f900000d64)
pci8086,3c0a (pciex, f900000d88)
pci15d9,720 (obp-device, f900000dac)
pci15d9,720 (obp-device, f900000dce)
pci15d9,720 (obp-device, f900000df0)
pci15d9,720 (obp-device, f900000e12)
pci15d9,720 (obp-device, f900000e34)
pci15d9,720 (obp-device, f900000e56)
pci15d9,720 (obp-device, f900000e78)
pci15d9,720 (obp-device, f900000e9a)
pci15d9,720 (obp-device, f900000ebc)
pci15d9,720 (obp-device, f900000edc)
pci15d9,720 (obp-device, f900000efc)
fw (acpirootnex, f900000f19)
sb (acpidevice, f900000f2f)
socket (acpicontainer, f900000f45)
cpu (cpu, f900000f5d)
cpu (cpu, f900000f9c)
cpu (cpu, f900000fdb)
cpu (cpu, f90000101a)
cpu (cpu, f900001059)
cpu (cpu, f900001098)
cpu (cpu, f9000010d7)
cpu (cpu, f900001116)
cpu (cpu, f900001155)
cpu (cpu, f900001194)
cpu (cpu, f9000011d3)
cpu (cpu, f900001212)
socket (acpicontainer, f900001251)
cpu (cpu, f900001269)
cpu (cpu, f9000012a8)
cpu (cpu, f9000012e7)
cpu (cpu, f900001326)
cpu (cpu, f900001365)
cpu (cpu, f9000013a4)
cpu (cpu, f9000013e3)
cpu (cpu, f900001422)
cpu (cpu, f900001461)
cpu (cpu, f9000014a0)
cpu (cpu, f9000014df)
cpu (cpu, f90000151e)
agpgart (agpgart, f900001580)
pseudo (devctl, f900001593)
zconsnex (devctl, f90000159c)
fct (admin, f9000015a5)
stmf (admin, f9000015ad)
stmf_sbd (admin, f9000015b5)
ppm (ppm, f9000015be)
power (power_button, f90000164e)
obp (picl, f900000059)
ioapics (ioapics, f900000760)
ioapic (ioapic, f900000766)
ioapic (ioapic, f90000076f)
used-resources (used-resources, f90000155d)
iscsi (iscsi, f900001567)
options (options, f900001589)
prtpicl-v.txt.gz
Description: GNU Zip compressed data
Loading kmdb... Welcome to kmdb kmdb: unable to determine terminal type: assuming `vt100' Loaded modules: [ unix krtld genunix ] [0]> ::bp pcieb`pcieb_attach [0]> :c SunOS Release 5.11 Version omnios-bc85f2d 64-bit Copyright (c) 1983, 2010, Oracle and/or its affiliates. All rights reserved. Loaded modules: [ scsi_vhci mac uppc zfs apix specfs pcplusmp cpu.generic ] kmdb: stop at pcieb`pcieb_attach kmdb: target stopped at: pcieb`pcieb_attach: pushq %rbp [0]> pcieb`pcieb_dbg_print/W 0x1 pcieb`pcieb_dbg_print: 0 = 0x1 [0]> ::delete 1 [0]> :c pcieb(0): pwrpcieb_pwr_disable: disabling PM pcieb(0): attachpcieb_intr_init: Attaching MSI handler pcieb(0): attachbdf 0xf7ad82a9: ddi_intr_get_nintrs: nintrs 8pcieb(0): attachVID:0x8086 DID:0x3c02 RID:0x7 bdf=0x8 pcieb(0): pwrINITCHILD: config regs setup for pci15d9,690@0 pcieb(1): pwrpcieb_pwr_disable: disabling PM pcieb(1): attachpcieb_intr_init: Attaching MSI handler pcieb(1): attachbdf 0xf7ad82a9: ddi_intr_get_nintrs: nintrs 16pcieb(1): attachVID:0x8086 DID:0x3c04 RID:0x7 bdf=0x10 pcieb(2): pwrpcieb_pwr_disable: disabling PM pcieb(2): attachpcieb_intr_init: Attaching MSI handler pcieb(2): attachbdf 0xf7ad82a9: ddi_intr_get_nintrs: nintrs 18pcieb(2): attachVID:0x8086 DID:0x3c06 RID:0x7 bdf=0x12 pcieb(3): pwrpcieb_pwr_disable: disabling PM pcieb(3): attachpcieb_intr_init: Attaching MSI handler pcieb(3): attachbdf 0xf7ad82a9: ddi_intr_get_nintrs: nintrs 19pcieb(3): attachVID:0x8086 DID:0x3c07 RID:0x7 bdf=0x13 pcieb(4): pwrpcieb_pwr_disable: disabling PM pcieb(4): attachpcieb_intr_init: Attaching MSI handler pcieb(4): attachbdf 0xf7ad82a9: ddi_intr_get_nintrs: nintrs 24pcieb(4): attachVID:0x8086 DID:0x3c08 RID:0x7 bdf=0x18 pcieb(6): pwrpcieb_pwr_disable: disabling PM pcieb(6): attachpcieb_intr_init: Attaching MSI handler pcieb(6): attachbdf 0xf7ad82a9: ddi_intr_get_nintrs: nintrs 136pcieb(6): attachVID:0x8086 DID:0x1d3e RID:0x6 bdf=0x88 pcieb(2): pwrINITCHILD: config regs setup for pci10e3,8113@0 pcieb(7): pwrpcieb_pwr_disable: disabling PM pcieb(7): attachpcieb_intr_init: Attaching INTx handler pcieb(7): attachVID:0x10e3 DID:0x8113 RID:0x1 bdf=0x700 pcieb(7): pwrINITCHILD: config regs setup for display@1 pcieb(3): pwrINITCHILD: config regs setup for pci15d9,1521@0 Hostname: head2 pcieb(3): pwrINITCHILD: config regs setup for pci15d9,1521@0,1 pcieb(1): pwrINITCHILD: config regs setup for pci15d9,10fb@0 pcieb(1): pwrINITCHILD: config regs setup for pci15d9,10fb@0,1 pcieb(4): pwrINITCHILD: config regs setup for pci103c,322d@0 head2 console login: ------------------------------------------- illumos-discuss Archives: https://www.listbox.com/member/archive/182180/=now RSS Feed: https://www.listbox.com/member/archive/rss/182180/21175430-2e6923be Modify Your Subscription: https://www.listbox.com/member/?member_id=21175430&id_secret=21175430-6a77cda4 Powered by Listbox: http://www.listbox.com
