Re: lspci not showing motherboard ethernet controller after PCIe card firmware update change from 32-bit to 64-bit BAR
On Wed, Sep 28, 2016 at 09:05:44AM -0500, Steve Kenton wrote: > I decided to experiment with /sys/bus/pci and the BIOS settings to try > and understand things better. > > The BIOS has a settings to enable/disable the on-board LAN. When the > Blackmagic card firmware is upgraded > bus[03] and hence the on-board LAN disappears, but the apparently unused > bus[02] is still visible and > the BIOS setting is still enabled. When the BIOS LAN settings is > disabled busses[02][03] and the on-board > LAN disappear. In both cases, the "disappeared" resources are not > restored by echo to /sys/.../rescan > > Why? What can the BIOS be doing to the chipset/busses[02][03] to make > them invisible to /sys manipulation? It can activate the reset line to the network chip making it never able to respond to anything. Or it could even power it off. And PCIe is point to point, so they are not really busses anymore. So at least turning off the network chip means no link on that point to point PCIe link and hence no "bus" there. > There is apparently something I do not understand about the H81 chipset > PCIe configuration. I have > been reading the Intel Series 8 / Series C220 Chipset documentation but > so far have come up empty. > > Anybody have cluebat for me? I wish. -- Len Sorensen
Re: lspci not showing motherboard ethernet controller after PCIe card firmware update change from 32-bit to 64-bit BAR
On Wed, Sep 28, 2016 at 09:05:44AM -0500, Steve Kenton wrote: > I decided to experiment with /sys/bus/pci and the BIOS settings to try > and understand things better. > > The BIOS has a settings to enable/disable the on-board LAN. When the > Blackmagic card firmware is upgraded > bus[03] and hence the on-board LAN disappears, but the apparently unused > bus[02] is still visible and > the BIOS setting is still enabled. When the BIOS LAN settings is > disabled busses[02][03] and the on-board > LAN disappear. In both cases, the "disappeared" resources are not > restored by echo to /sys/.../rescan > > Why? What can the BIOS be doing to the chipset/busses[02][03] to make > them invisible to /sys manipulation? It can activate the reset line to the network chip making it never able to respond to anything. Or it could even power it off. And PCIe is point to point, so they are not really busses anymore. So at least turning off the network chip means no link on that point to point PCIe link and hence no "bus" there. > There is apparently something I do not understand about the H81 chipset > PCIe configuration. I have > been reading the Intel Series 8 / Series C220 Chipset documentation but > so far have come up empty. > > Anybody have cluebat for me? I wish. -- Len Sorensen
Re: lspci not showing motherboard ethernet controller after PCIe card firmware update change from 32-bit to 64-bit BAR
On 09/26/2016 10:50 AM, Lennart Sorensen wrote: > Well pci=bios is 32 bit only, so that doesn't work, pci=bfs was probably > supposed to be pci=bfsort. It sure looks like the bios fails to allocate > the network device when the other device is 64 bit. Sure looks like > a bios bug. I see nothing indicating that the address space ran out. > It looks fine. > > You could try: > > pci=realloc > pci=assign-busses > pci=bfsort (the one I think you were trying to do). > > I certainly wouldn't be surprised if the only thing ever tested in that > board was a graphics card. > > Of course your network card is also 64 bit bar (when seen), so the system > can work with such. Makes one wonder if the bios has a bus, or the H81 > has a limit that only one of the ethernet and the add in card can use > 64 bit bar at a time. That would seem weird of course. > I decided to experiment with /sys/bus/pci and the BIOS settings to try and understand things better. The BIOS has a settings to enable/disable the on-board LAN. When the Blackmagic card firmware is upgraded bus[03] and hence the on-board LAN disappears, but the apparently unused bus[02] is still visible and the BIOS setting is still enabled. When the BIOS LAN settings is disabled busses[02][03] and the on-board LAN disappear. In both cases, the "disappeared" resources are not restored by echo to /sys/.../rescan Why? What can the BIOS be doing to the chipset/busses[02][03] to make them invisible to /sys manipulation? There is apparently something I do not understand about the H81 chipset PCIe configuration. I have been reading the Intel Series 8 / Series C220 Chipset documentation but so far have come up empty. Anybody have cluebat for me? Steve Kenton = Here is the lspci tree output as configured by the BIOS with the on-board LAN enabled Using echo to various /sys files successfully removes and restores(rescan) the Blackmagic card, the Realtek LAN as well as busses[01][02][03] -[:00]-+-00.0 Intel Corporation 4th Gen Core Processor DRAM Controller [8086:0c00] +-01.0-[01]00.0 Blackmagic Design Intensity Pro 4K [bdbd:a139] +-02.0 Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller [8086:0412] +-03.0 Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller [8086:0c0c] +-14.0 Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI [8086:8c31] +-16.0 Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 [8086:8c3a] +-1a.0 Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 [8086:8c2d] +-1b.0 Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller [8086:8c20] +-1c.0-[02]-- +-1c.5-[03]00.0 Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] +-1d.0 Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 [8086:8c26] +-1f.0 Intel Corporation C220 Series Chipset Family H81 Express LPC Controller [8086:8c5c] +-1f.2 Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] [8086:8c02] \-1f.3 Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller [8086:8c22] = Here is the lspci tree output as configured by the BIOS with the on-board LAN disabled Using echo to various /sys files I am unable to restore busses[02][03] so I cannot even attempt to restore the Realtek LAN -[:00]-+-00.0 Intel Corporation 4th Gen Core Processor DRAM Controller [8086:0c00] +-01.0-[01]00.0 Blackmagic Design Intensity Pro 4K [bdbd:a139] +-02.0 Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller [8086:0412] +-03.0 Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller [8086:0c0c] +-14.0 Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI [8086:8c31] +-16.0 Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 [8086:8c3a] +-1a.0 Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 [8086:8c2d] +-1b.0 Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller [8086:8c20] +-1d.0 Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 [8086:8c26] +-1f.0 Intel Corporation C220 Series Chipset Family H81 Express LPC Controller [8086:8c5c] +-1f.2 Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] [8086:8c02] \-1f.3 Intel Corporation 8
Re: lspci not showing motherboard ethernet controller after PCIe card firmware update change from 32-bit to 64-bit BAR
On 09/26/2016 10:50 AM, Lennart Sorensen wrote: > Well pci=bios is 32 bit only, so that doesn't work, pci=bfs was probably > supposed to be pci=bfsort. It sure looks like the bios fails to allocate > the network device when the other device is 64 bit. Sure looks like > a bios bug. I see nothing indicating that the address space ran out. > It looks fine. > > You could try: > > pci=realloc > pci=assign-busses > pci=bfsort (the one I think you were trying to do). > > I certainly wouldn't be surprised if the only thing ever tested in that > board was a graphics card. > > Of course your network card is also 64 bit bar (when seen), so the system > can work with such. Makes one wonder if the bios has a bus, or the H81 > has a limit that only one of the ethernet and the add in card can use > 64 bit bar at a time. That would seem weird of course. > I decided to experiment with /sys/bus/pci and the BIOS settings to try and understand things better. The BIOS has a settings to enable/disable the on-board LAN. When the Blackmagic card firmware is upgraded bus[03] and hence the on-board LAN disappears, but the apparently unused bus[02] is still visible and the BIOS setting is still enabled. When the BIOS LAN settings is disabled busses[02][03] and the on-board LAN disappear. In both cases, the "disappeared" resources are not restored by echo to /sys/.../rescan Why? What can the BIOS be doing to the chipset/busses[02][03] to make them invisible to /sys manipulation? There is apparently something I do not understand about the H81 chipset PCIe configuration. I have been reading the Intel Series 8 / Series C220 Chipset documentation but so far have come up empty. Anybody have cluebat for me? Steve Kenton = Here is the lspci tree output as configured by the BIOS with the on-board LAN enabled Using echo to various /sys files successfully removes and restores(rescan) the Blackmagic card, the Realtek LAN as well as busses[01][02][03] -[:00]-+-00.0 Intel Corporation 4th Gen Core Processor DRAM Controller [8086:0c00] +-01.0-[01]00.0 Blackmagic Design Intensity Pro 4K [bdbd:a139] +-02.0 Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller [8086:0412] +-03.0 Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller [8086:0c0c] +-14.0 Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI [8086:8c31] +-16.0 Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 [8086:8c3a] +-1a.0 Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 [8086:8c2d] +-1b.0 Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller [8086:8c20] +-1c.0-[02]-- +-1c.5-[03]00.0 Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] +-1d.0 Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 [8086:8c26] +-1f.0 Intel Corporation C220 Series Chipset Family H81 Express LPC Controller [8086:8c5c] +-1f.2 Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] [8086:8c02] \-1f.3 Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller [8086:8c22] = Here is the lspci tree output as configured by the BIOS with the on-board LAN disabled Using echo to various /sys files I am unable to restore busses[02][03] so I cannot even attempt to restore the Realtek LAN -[:00]-+-00.0 Intel Corporation 4th Gen Core Processor DRAM Controller [8086:0c00] +-01.0-[01]00.0 Blackmagic Design Intensity Pro 4K [bdbd:a139] +-02.0 Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller [8086:0412] +-03.0 Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller [8086:0c0c] +-14.0 Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI [8086:8c31] +-16.0 Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 [8086:8c3a] +-1a.0 Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 [8086:8c2d] +-1b.0 Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller [8086:8c20] +-1d.0 Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 [8086:8c26] +-1f.0 Intel Corporation C220 Series Chipset Family H81 Express LPC Controller [8086:8c5c] +-1f.2 Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] [8086:8c02] \-1f.3 Intel Corporation 8
Re: lspci not showing motherboard ethernet controller after PCIe card firmware update change from 32-bit to 64-bit BAR
On Sat, Sep 24, 2016 at 07:24:14PM -0500, Steve Kenton wrote: > I have two different systems with Blackmagic Design Intensity Pro-4K > cards. One uses an ECS H81H3-I motherboard and the other uses a Gigabyte > GA-H110N motherboard. Previously both were working properly. When I > upgraded the Blackmagic Design desktopvideo package from version 10.7 to > version 10.8 it included an update of the firmware on the cards from > version 0x85 to 0xbd. After the upgrade the Gigabyte-motherboard system > continues to work properly, but the ECS-motherboard system no longer > shows the motherboard ethernet controller with lspci. Downgrading the > firmware makes the ethernet controller reappear with lspci. Upgrading > again makes it disappear again. Attached are the output of lscpi -vv > for each version of the firmware on the ECS-motherboard system. > Upgrading to the most recent ECS-motherboard bios did not have any effect. > > There are a number of differences in the two lspci outputs but > based on conversations I've seen on LKM, which I peruse but am not > subscribed to, I suspect that this is the important one: > > lspci-Intensity-Pro-4K-10.7-firmware.txt:Region 0: Memory at > f7d0 (32-bit, non-prefetchable) [size=1M] > lspci-Intensity-Pro-4K-10.8-firmware.txt:Region 0: Memory at > f7c0 (64-bit, non-prefetchable) [size=1M] > > The Engineers at Blackmagic Design have confirmed that the change to a > 64-bit BAR with firmware version 0xbd was intentional and correct. And > to be clear, the Blackmagic cards continue to work properly. The issue > is the motherboard ethernet controller disappearing after upgrading the > Blackmagic cards to new firmware with a 64-bit BAR. Note, however, that > 64-bit BAR is below 4GB with the high four bytes being all zeros. After > the firmware update it's like there is "no room at the inn" for the > motherboard ethernet controller so it never gets enabled. I think we can > all agree that it's a likely a problem with the PC motherboard bios. I'm > hoping to find a workaround to allow me to configure a number of > existing ECS-motherboard systems in the field for use with the new firmware. > > I normally run Ubuntu 14.04 LTS and tried the most recent 4.4.0-38 > kernel but then I noticed that the Ubuntu 16.10 daily build ISO switched > to kernel 4.8-RC-something on Sept 21st. So I booted that > livedvd and tried it to see if 4.8-RC included some recent PCIe > 32-bit/64-bit resource fix but sadly the results were the same. With > firmware 0xbd, the ethernet controller was still missing from the lspci > output. I poked around with various of the kernel pci= parameters such > as bios and nobios to no avail, but I really am out of my depth here. > > I'm available to run tests if anyone has any suggestions. Well pci=bios is 32 bit only, so that doesn't work, pci=bfs was probably supposed to be pci=bfsort. It sure looks like the bios fails to allocate the network device when the other device is 64 bit. Sure looks like a bios bug. I see nothing indicating that the address space ran out. It looks fine. You could try: pci=realloc pci=assign-busses pci=bfsort (the one I think you were trying to do). I certainly wouldn't be surprised if the only thing ever tested in that board was a graphics card. Of course your network card is also 64 bit bar (when seen), so the system can work with such. Makes one wonder if the bios has a bus, or the H81 has a limit that only one of the ethernet and the add in card can use 64 bit bar at a time. That would seem weird of course. -- Len Sorensen
Re: lspci not showing motherboard ethernet controller after PCIe card firmware update change from 32-bit to 64-bit BAR
On Sat, Sep 24, 2016 at 07:24:14PM -0500, Steve Kenton wrote: > I have two different systems with Blackmagic Design Intensity Pro-4K > cards. One uses an ECS H81H3-I motherboard and the other uses a Gigabyte > GA-H110N motherboard. Previously both were working properly. When I > upgraded the Blackmagic Design desktopvideo package from version 10.7 to > version 10.8 it included an update of the firmware on the cards from > version 0x85 to 0xbd. After the upgrade the Gigabyte-motherboard system > continues to work properly, but the ECS-motherboard system no longer > shows the motherboard ethernet controller with lspci. Downgrading the > firmware makes the ethernet controller reappear with lspci. Upgrading > again makes it disappear again. Attached are the output of lscpi -vv > for each version of the firmware on the ECS-motherboard system. > Upgrading to the most recent ECS-motherboard bios did not have any effect. > > There are a number of differences in the two lspci outputs but > based on conversations I've seen on LKM, which I peruse but am not > subscribed to, I suspect that this is the important one: > > lspci-Intensity-Pro-4K-10.7-firmware.txt:Region 0: Memory at > f7d0 (32-bit, non-prefetchable) [size=1M] > lspci-Intensity-Pro-4K-10.8-firmware.txt:Region 0: Memory at > f7c0 (64-bit, non-prefetchable) [size=1M] > > The Engineers at Blackmagic Design have confirmed that the change to a > 64-bit BAR with firmware version 0xbd was intentional and correct. And > to be clear, the Blackmagic cards continue to work properly. The issue > is the motherboard ethernet controller disappearing after upgrading the > Blackmagic cards to new firmware with a 64-bit BAR. Note, however, that > 64-bit BAR is below 4GB with the high four bytes being all zeros. After > the firmware update it's like there is "no room at the inn" for the > motherboard ethernet controller so it never gets enabled. I think we can > all agree that it's a likely a problem with the PC motherboard bios. I'm > hoping to find a workaround to allow me to configure a number of > existing ECS-motherboard systems in the field for use with the new firmware. > > I normally run Ubuntu 14.04 LTS and tried the most recent 4.4.0-38 > kernel but then I noticed that the Ubuntu 16.10 daily build ISO switched > to kernel 4.8-RC-something on Sept 21st. So I booted that > livedvd and tried it to see if 4.8-RC included some recent PCIe > 32-bit/64-bit resource fix but sadly the results were the same. With > firmware 0xbd, the ethernet controller was still missing from the lspci > output. I poked around with various of the kernel pci= parameters such > as bios and nobios to no avail, but I really am out of my depth here. > > I'm available to run tests if anyone has any suggestions. Well pci=bios is 32 bit only, so that doesn't work, pci=bfs was probably supposed to be pci=bfsort. It sure looks like the bios fails to allocate the network device when the other device is 64 bit. Sure looks like a bios bug. I see nothing indicating that the address space ran out. It looks fine. You could try: pci=realloc pci=assign-busses pci=bfsort (the one I think you were trying to do). I certainly wouldn't be surprised if the only thing ever tested in that board was a graphics card. Of course your network card is also 64 bit bar (when seen), so the system can work with such. Makes one wonder if the bios has a bus, or the H81 has a limit that only one of the ethernet and the add in card can use 64 bit bar at a time. That would seem weird of course. -- Len Sorensen
