Re: [coreboot] unstable AMD Fam10h boot
On Mon, Sep 14, 2009 at 7:51 PM, Bao, Zheng zheng@amd.com wrote: Are you sure the pci functions will cover the case that the address is more than 0x100? It should, unless you know something I don't (bug?). Using the MMIO config access is the preferred method since it enforces the ordering. See 2.11 Configuration Space in the BKDG. Marc -- http://marcjonesconsulting.com -- coreboot mailing list: coreboot@coreboot.org http://www.coreboot.org/mailman/listinfo/coreboot
Re: [coreboot] unstable AMD Fam10h boot
I am not sure what I was thinking last night. This is really simple... There is no address manipulation to be done before calling the coreboot pci functions. Thanks Patrick Marc -- http://marcjonesconsulting.com pci_rw.patch Description: Binary data -- coreboot mailing list: coreboot@coreboot.org http://www.coreboot.org/mailman/listinfo/coreboot
Re: [coreboot] unstable AMD Fam10h boot
On Mon, Sep 14, 2009 at 10:45 AM, Marc Jones marcj...@gmail.com wrote: I am not sure what I was thinking last night. This is really simple... There is no address manipulation to be done before calling the coreboot pci functions. Thanks Patrick It looks like GetNB and SetNB should die. Is there a purpose for having the extra indirection? Thanks, Myles -- coreboot mailing list: coreboot@coreboot.org http://www.coreboot.org/mailman/listinfo/coreboot
Re: [coreboot] unstable AMD Fam10h boot
Am Montag, den 14.09.2009, 10:45 -0600 schrieb Marc Jones: I am not sure what I was thinking last night. This is really simple... There is no address manipulation to be done before calling the coreboot pci functions. Thanks Patrick The board boots much more reliable now, thank you! Acked-by: Patrick Georgi patrick.geo...@coresystems.de -- coreboot mailing list: coreboot@coreboot.org http://www.coreboot.org/mailman/listinfo/coreboot
Re: [coreboot] unstable AMD Fam10h boot
Myles Watson wrote: It looks like GetNB and SetNB should die. +1 //Peter -- coreboot mailing list: coreboot@coreboot.org http://www.coreboot.org/mailman/listinfo/coreboot
Re: [coreboot] unstable AMD Fam10h boot
On Mon, Sep 14, 2009 at 10:52 AM, Peter Stuge pe...@stuge.se wrote: Myles Watson wrote: It looks like GetNB and SetNB should die. +1 I guess it should. The only reason to keep it is for easily diffing against the AMD AGESA reference code, but I am not certain if or when that will happen again. Marc -- http://marcjonesconsulting.com -- coreboot mailing list: coreboot@coreboot.org http://www.coreboot.org/mailman/listinfo/coreboot
Re: [coreboot] unstable AMD Fam10h boot
On Mon, Sep 14, 2009 at 10:50 AM, Patrick Georgi patr...@georgi-clan.de wrote: Am Montag, den 14.09.2009, 10:45 -0600 schrieb Marc Jones: I am not sure what I was thinking last night. This is really simple... There is no address manipulation to be done before calling the coreboot pci functions. Thanks Patrick The board boots much more reliable now, thank you! Acked-by: Patrick Georgi patrick.geo...@coresystems.de Thanks Patrick. r4633 -- http://marcjonesconsulting.com -- coreboot mailing list: coreboot@coreboot.org http://www.coreboot.org/mailman/listinfo/coreboot
Re: [coreboot] unstable AMD Fam10h boot
Are you sure the pci functions will cover the case that the address is more than 0x100? Zheng -Original Message- From: coreboot-boun...@coreboot.org [mailto:coreboot-boun...@coreboot.org] On Behalf Of Marc Jones Sent: Tuesday, September 15, 2009 12:45 AM To: ron minnich Cc: coreboot@coreboot.org Subject: Re: [coreboot] unstable AMD Fam10h boot I am not sure what I was thinking last night. This is really simple... There is no address manipulation to be done before calling the coreboot pci functions. Thanks Patrick Marc -- http://marcjonesconsulting.com -- coreboot mailing list: coreboot@coreboot.org http://www.coreboot.org/mailman/listinfo/coreboot
Re: [coreboot] unstable AMD Fam10h boot
On Tue, Sep 8, 2009 at 11:02 AM, Marc Jones marcj...@gmail.com wrote:
On Sun, Sep 6, 2009 at 4:32 PM, ron minnichrminn...@gmail.com wrote:
The way I see it the memory setup and SMP support in CAR are two very
different issues.
This bug is totally my fault...
Yes, Memory setup and SMP CAR are two different issues. The SMP setup
happens during CAR is to setup microcode, HT and FIDVID prior to the
PLL reset and memory setup.
All the SMP PCI config space access should be MMIO. It is the first
thing that is enabled in CPU init in set_pci_mmio_conf_reg().
The bug is that I mixed a mem setup function in with SMP setup by
using mctGetLogicalCPUID() which uses Get_NB32. As pointed out, the
GET_NB32 is a cf8/cfc function. The mct code ported from AGESA assumes
that it is running on the BSP only and uses cf8/cfc. (historical
k8 bug I think)
I think that I should change the mct PCI config functions to call the
coreboot pci_read_config32 functions that handle MMIO vs cfc/cf8
nicely. This should future proof mct functions in CAR and a step
toward SMP memory setup.
Some of that mct code PCI config space code is a little funny (ok, a
lot funny), so it will take a little care. I should be able work patch
in a couple of days.
Here is a patch that fixes the cf8 config access. Not complicated like
I initially recalled. Thanks to Ralf for pointing out the bug.
This needs testing. Anyone?
Signed-off-by: Marc Jones marcj...@gmail.com
Thanks,
Marc
--
http://marcjonesconsulting.com
Use the coreboot pci config read/write functions instead of direct cf8/cfc access. The fam10 pci functions will use mmio and not have SMP pci access issues.
Signed-off-by: Marc Jones marcj...@gmail.com↔
Index: coreboot-v2/src/northbridge/amd/amdmct/mct/mct_d.c
===
--- coreboot-v2.orig/src/northbridge/amd/amdmct/mct/mct_d.c 2009-09-13 18:01:54.0 -0600
+++ coreboot-v2/src/northbridge/amd/amdmct/mct/mct_d.c 2009-09-13 18:09:08.0 -0600
@@ -2474,10 +2474,8 @@
{
u32 addr;
- addr = (dev4) | (reg 0xFF) | ((reg 0xf00)16);
- outl((131) | (addr ~3), 0xcf8);
-
- return inl(0xcfc);
+ addr = ((dev4) | (reg 0xFF) | ((reg 0xf00)16)) ~3;
+ return pci_read_config32(addr);
}
@@ -2485,9 +2483,8 @@
{
u32 addr;
- addr = (dev4) | (reg 0xFF) | ((reg 0xf00)16);
- outl((131) | (addr ~3), 0xcf8);
- outl(val, 0xcfc);
+ addr = ((dev4) | (reg 0xFF) | ((reg 0xf00)16)) ~3;
+ pci_write_config32(addr);
}
--
coreboot mailing list: coreboot@coreboot.org
http://www.coreboot.org/mailman/listinfo/coreboot
Re: [coreboot] unstable AMD Fam10h boot
On Sun, Sep 13, 2009 at 06:46:38PM -0600, Marc Jones wrote: Here is a patch that fixes the cf8 config access. Not complicated like I initially recalled. Thanks to Ralf for pointing out the bug. This needs testing. Anyone? We've got a couple H8DME/fam10 boxes coming this week, so I should be able to test this in a couple days. Will do as soon as we get the hardware. Thanks, Ward. -- Ward Vandewege w...@fsf.org Free Software Foundation - Senior Systems Administrator -- coreboot mailing list: coreboot@coreboot.org http://www.coreboot.org/mailman/listinfo/coreboot
Re: [coreboot] unstable AMD Fam10h boot
On Sun, Sep 6, 2009 at 4:32 PM, ron minnichrminn...@gmail.com wrote: The way I see it the memory setup and SMP support in CAR are two very different issues. This bug is totally my fault... Yes, Memory setup and SMP CAR are two different issues. The SMP setup happens during CAR is to setup microcode, HT and FIDVID prior to the PLL reset and memory setup. All the SMP PCI config space access should be MMIO. It is the first thing that is enabled in CPU init in set_pci_mmio_conf_reg(). The bug is that I mixed a mem setup function in with SMP setup by using mctGetLogicalCPUID() which uses Get_NB32. As pointed out, the GET_NB32 is a cf8/cfc function. The mct code ported from AGESA assumes that it is running on the BSP only and uses cf8/cfc. (historical k8 bug I think) I think that I should change the mct PCI config functions to call the coreboot pci_read_config32 functions that handle MMIO vs cfc/cf8 nicely. This should future proof mct functions in CAR and a step toward SMP memory setup. Some of that mct code PCI config space code is a little funny (ok, a lot funny), so it will take a little care. I should be able work patch in a couple of days. Marc -- http://marcjonesconsulting.com -- coreboot mailing list: coreboot@coreboot.org http://www.coreboot.org/mailman/listinfo/coreboot
Re: [coreboot] unstable AMD Fam10h boot
Peter Stuge wrote: Stefan Reinauer wrote: RAM init working concurrently with multiple memory controllers - they're on PCI, right? Yes. But why would it not be completely sufficient to set up all ram controllers in the system from the BSP? The big coreboot SMP win is with ECC scrubbing, right? Yes, but that does not happen until we're in stage2. It's not really part of memory init. Does that involve some PCI config space accesses to the memory controllers? I don't think so. -- coreboot mailing list: coreboot@coreboot.org http://www.coreboot.org/mailman/listinfo/coreboot
Re: [coreboot] unstable AMD Fam10h boot
ron minnich wrote: 2. There are techniques from the oldest days of PCI (cf8/cfc) that we use that don't work well in a multicore world. Since there is only one set of PCI devices, I wonder what the benefit would be to penetrate them from all CPU cores or what would even cause that as a requirement. 3. we need smp style locking code for printk. We have. But not in raminit. Stefan -- coreboot mailing list: coreboot@coreboot.org http://www.coreboot.org/mailman/listinfo/coreboot
Re: [coreboot] unstable AMD Fam10h boot
Stefan Reinauer wrote: Since there is only one set of PCI devices, I wonder what the benefit would be to penetrate them from all CPU cores or what would even cause that as a requirement. RAM init working concurrently with multiple memory controllers - they're on PCI, right? //Peter -- coreboot mailing list: coreboot@coreboot.org http://www.coreboot.org/mailman/listinfo/coreboot
Re: [coreboot] unstable AMD Fam10h boot
Peter Stuge wrote: Stefan Reinauer wrote: Since there is only one set of PCI devices, I wonder what the benefit would be to penetrate them from all CPU cores or what would even cause that as a requirement. RAM init working concurrently with multiple memory controllers - they're on PCI, right? Yes. But why would it not be completely sufficient to set up all ram controllers in the system from the BSP? Or, put differently. We're smart and we fix the PCI problem. Then we suddenly notice that that is not enough. because a PCI operation is by no means an atomic operation. We're going to have to add another layer of locking on top of that, for example for SMBUS access, which might involve PCI access. I'm saying we're opening a can of worms here, and unless we really like to go fishing we should close it again and walk in the dry. Stefan -- coreboot mailing list: coreboot@coreboot.org http://www.coreboot.org/mailman/listinfo/coreboot
Re: [coreboot] unstable AMD Fam10h boot
On Tue, Sep 1, 2009 at 2:15 PM, Ralf Grosse Boergerral...@gmail.com wrote:
Hi,
this a reply to the following message:
http://www.coreboot.org/pipermail/coreboot/2009-August/051629.html
[I am not subscribed to this list.]
The sporadic boot problems (FIXME! CPU Version unknown or not supported!)
are caused by a race condition in Get_NB32().
This function performs a read operation to the PCI configuration space via
port CF8/CFC.
u32 Get_NB32(u32 dev, u32 reg)
{
u32 addr;
addr = (dev4) | (reg 0xFF) | ((reg 0xf00)16);
outl((131) | (addr ~3), 0xcf8);
return inl(0xcfc);
}
As ports CF8/CFC are shared across cores (maybe even sockets?) concurrent
accesses from different cores may yield random results.
OK, let's start this discussion again.
Can we at least answer this question. Ports CF8/CFC are shared across
- sockets
- cores
I am betting they are not shared across sockets, and would be
surprised if they are shared across cores but am
willing to believe it.
Anybody?
ron
--
coreboot mailing list: coreboot@coreboot.org
http://www.coreboot.org/mailman/listinfo/coreboot
Re: [coreboot] unstable AMD Fam10h boot
Stefan Reinauer wrote: RAM init working concurrently with multiple memory controllers - they're on PCI, right? Yes. But why would it not be completely sufficient to set up all ram controllers in the system from the BSP? The big coreboot SMP win is with ECC scrubbing, right? Does that involve some PCI config space accesses to the memory controllers? Is it simpler to create locking for PCI accesses, or to split out the part of RAM init which sets up MCs into the BSP (as opposed to keeping the code for each MC running on that same core)? I'm saying we're opening a can of worms here, and unless we really like to go fishing we should close it again and walk in the dry. I'm thinking we're already hooked on that tasty fish? (By fish I mean SMP memory init.) //Peter -- coreboot mailing list: coreboot@coreboot.org http://www.coreboot.org/mailman/listinfo/coreboot
Re: [coreboot] unstable AMD Fam10h boot
The way I see it the memory setup and SMP support in CAR are two very different issues. BSP can do its own memory. Once that memory is set up the APs can use it. Thus, the APs can have working memory when they do their RAM setup. In other words, BSP does RAM setup in CAR APs can do RAM setup with working RAM -- they just use the BSP RAM, which is working. The K8 code hints of this model, and, when I did my trial code for V3, this is how I set it up to work. Hence, we can do SMP memory setup as long as the BSP sets up its own memory before it starts up the APs. We are really talking about SMP in CAR, which seems like a much harder issue. Make sense? Something I'm missing? ron -- coreboot mailing list: coreboot@coreboot.org http://www.coreboot.org/mailman/listinfo/coreboot
Re: [coreboot] unstable AMD Fam10h boot
ron minnich wrote: So, what I'm trying to say: - we have a problem on fam10h - it seems to be a non-smp-safe function doing a config cycle - there are two ways to eliminate the problem o write a fam10 version of that function that will use MMCONF (will work on all later CPUs) o modify old function by adding a lock (i.e. stick with legacy mechanism for older CPUs) Another idea would be to get rid of SMP setup in CAR stage. It sounds highly funky to me anyways. - Why are we doing this anyways? o Is there a reason? o No other SMP system except K10 does this. * How many ms do we benefit from that? (Honest question). Any at all? Stefan -- coreboot mailing list: coreboot@coreboot.org http://www.coreboot.org/mailman/listinfo/coreboot
Re: [coreboot] unstable AMD Fam10h boot
On Sat, Sep 5, 2009 at 10:36 AM, Stefan Reinauerste...@coresystems.de wrote: Another idea would be to get rid of SMP setup in CAR stage. It sounds highly funky to me anyways. - Why are we doing this anyways? o Is there a reason? o No other SMP system except K10 does this. * How many ms do we benefit from that? (Honest question). Any at all? This may fix one problem, but it does not fix the general problem: using cf8/cfc is not going to be safe on multiple cores, from my understanding. ron -- coreboot mailing list: coreboot@coreboot.org http://www.coreboot.org/mailman/listinfo/coreboot
Re: [coreboot] unstable AMD Fam10h boot
On Sat, Sep 05, 2009 at 10:46:57AM -0700, ron minnich wrote: On Sat, Sep 5, 2009 at 10:36 AM, Stefan Reinauerste...@coresystems.de wrote: Another idea would be to get rid of SMP setup in CAR stage. It sounds highly funky to me anyways. - Why are we doing this anyways? o Is there a reason? o No other SMP system except K10 does this. * How many ms do we benefit from that? (Honest question). Any at all? This may fix one problem, but it does not fix the general problem: using cf8/cfc is not going to be safe on multiple cores, from my understanding. Not to complicate matters even further, but since we are talking about locking - will any of this improve the 'many cores talking to serial at once' problem? Thanks, Ward. -- Ward Vandewege w...@fsf.org Free Software Foundation - Senior Systems Administrator -- coreboot mailing list: coreboot@coreboot.org http://www.coreboot.org/mailman/listinfo/coreboot
Re: [coreboot] unstable AMD Fam10h boot
there's at least three problems here :-) 1. do we want to do SMP of any kind in CAR? I think not, but I'd like to hear Marc's opinion 2. There are techniques from the oldest days of PCI (cf8/cfc) that we use that don't work well in a multicore world. MMCONF, it seems, can help here. Using MMCONF for config cycles may only work for newer CPUs, but we really ought to move to something that doesn't require a shared resource such as cf8/cfc for more modern CPUs. 3. we need smp style locking code for printk. Any more :-) ron -- coreboot mailing list: coreboot@coreboot.org http://www.coreboot.org/mailman/listinfo/coreboot
Re: [coreboot] unstable AMD Fam10h boot
On Tue, Sep 1, 2009 at 2:15 PM, Ralf Grosse Boergerral...@gmail.com wrote:
Hi,
this a reply to the following message:
http://www.coreboot.org/pipermail/coreboot/2009-August/051629.html
[I am not subscribed to this list.]
The sporadic boot problems (FIXME! CPU Version unknown or not supported!)
are caused by a race condition in Get_NB32().
This function performs a read operation to the PCI configuration space via
port CF8/CFC.
u32 Get_NB32(u32 dev, u32 reg)
{
u32 addr;
addr = (dev4) | (reg 0xFF) | ((reg 0xf00)16);
outl((131) | (addr ~3), 0xcf8);
return inl(0xcfc);
}
As ports CF8/CFC are shared across cores (maybe even sockets?) concurrent
accesses from different cores may yield random results.
I would be surprised were they shared across sockets but ...
I'm realizing I have no clue how config cycles work on Opteron. I just
assumed this cf8/cfc cycle was magically converted inside the cpu into
an HT cycle of some sort, and that cycle was routed via the config
space maps in the NB. But ... can someone inform me on how this really
works? Is my picture even close?
Is there some better way on fam10 to do config cycles that is more
multi-core friendly? It seems odd that we are still locked into this
cf8/cfc stuff.
ron
--
coreboot mailing list: coreboot@coreboot.org
http://www.coreboot.org/mailman/listinfo/coreboot
Re: [coreboot] unstable AMD Fam10h boot
As ports CF8/CFC are shared across cores (maybe even sockets?) concurrent accesses from different cores may yield random results. I would be surprised were they shared across sockets but ... I'm realizing I have no clue how config cycles work on Opteron. I just assumed this cf8/cfc cycle was magically converted inside the cpu into an HT cycle of some sort, and that cycle was routed via the config space maps in the NB. But ... can someone inform me on how this really works? Is my picture even close? I don't know how the conversion works exactly, or where it takes place, but the HT packet is a read or a write to 0xFD.FE00. + an offset for the UnitID(pci device number). So, for device 7 on the bus and config register 0x14, you get 0xFDFE003814. You don't have to worry about bus numbers because they get taken care of based on the HT chain the to which the packet is routed. Based on that, I would say it's not shared across sockets, but it definitely could be shared across cores. Is there some better way on fam10 to do config cycles that is more multi-core friendly? It seems odd that we are still locked into this cf8/cfc stuff. Isn't there a way to do MMCONF cycles from the NB? Myles -- coreboot mailing list: coreboot@coreboot.org http://www.coreboot.org/mailman/listinfo/coreboot
Re: [coreboot] unstable AMD Fam10h boot
On Fri, Sep 4, 2009 at 9:57 AM, Myles Watsonmyle...@gmail.com wrote: As ports CF8/CFC are shared across cores (maybe even sockets?) concurrent accesses from different cores may yield random results. I would be surprised were they shared across sockets but ... I'm realizing I have no clue how config cycles work on Opteron. I just assumed this cf8/cfc cycle was magically converted inside the cpu into an HT cycle of some sort, and that cycle was routed via the config space maps in the NB. But ... can someone inform me on how this really works? Is my picture even close? I don't know how the conversion works exactly, or where it takes place, but the HT packet is a read or a write to 0xFD.FE00. + an offset for the UnitID(pci device number). So, for device 7 on the bus and config register 0x14, you get 0xFDFE003814. You don't have to worry about bus numbers because they get taken care of based on the HT chain the to which the packet is routed. This is a memory read/write? Based on that, I would say it's not shared across sockets, but it definitely could be shared across cores. Isn't there a way to do MMCONF cycles from the NB? Marc? If there were, it would make sense to convert the code to use these, instead of trying to make cf8/cfc SMP-safe. ron -- coreboot mailing list: coreboot@coreboot.org http://www.coreboot.org/mailman/listinfo/coreboot
Re: [coreboot] unstable AMD Fam10h boot
On Fri, Sep 4, 2009 at 10:07 AM, Rudolf Marekr.ma...@assembler.cz wrote: I think the only proper way is to do type1 and maybe some locking is necessary. but our thread subject is unstable fam10h. Given that fam10h is the problem, and that it supports MMCONF, why not make new versions of the functions for processors that have MMCONF and use them on those processors? We've never seen this kind of problem on K8 AFAIK. We can continue to use the old functions on those old CPUs. So, what I'm trying to say: - we have a problem on fam10h - it seems to be a non-smp-safe function doing a config cycle - there are two ways to eliminate the problem o write a fam10 version of that function that will use MMCONF (will work on all later CPUs) o modify old function by adding a lock (i.e. stick with legacy mechanism for older CPUs) I just can't see a good reason to stick with the type 1 access when the fam10h and, presumably all later families, will support MMCONF. The cf8/cfc is a 15-year-old idea (at least) that predates smp and multicore. We should be trying to eliminate that old mechanism whenever we can (at least it seems that way to me). It is the cf8/cfc mechanism that is the problem, not the lack of locking. ron -- coreboot mailing list: coreboot@coreboot.org http://www.coreboot.org/mailman/listinfo/coreboot
Re: [coreboot] unstable AMD Fam10h boot
Hi all, I think I just found an answer: 2.11 Configuration Space This BKDG chapter suggest that there is a MSR which can be used to do MMIO accesses. MSRC001_0058 MMIO Configuration Base Address Register Note that all cores should have this addr. Also it seems that this is what\ it should be used in ACPI instead of the NB MCFG aperture. Rudolf -- coreboot mailing list: coreboot@coreboot.org http://www.coreboot.org/mailman/listinfo/coreboot
Re: [coreboot] unstable AMD Fam10h boot
Isn't there a way to do MMCONF cycles from the NB? No, I think only newer fam10h cpus can do that. http://lkml.org/lkml/2007/12/21/134 So, the K8 internal devices are not accessible with the mmconf aperture which is NB dependent anyway. That post says that it may work for newer fam10h cpus. I think this is related: F1x[EC:E0] Configuration Map Registers and DisCohLdtCfg: disable coherent link configuration accesses. I think linux has some test which test what devices can be accessed through type1 and what it sees on mmconf. http://www.x86-64.org/pipermail/discuss/2005-December/007371.html I think the only proper way is to do type1 and maybe some locking is necessary. Rudolf -- coreboot mailing list: coreboot@coreboot.org http://www.coreboot.org/mailman/listinfo/coreboot
Re: [coreboot] unstable AMD Fam10h boot
I don't know how the conversion works exactly, or where it takes place, but the HT packet is a read or a write to 0xFD.FE00. + an offset for the UnitID(pci device number). So, for device 7 on the bus and config register 0x14, you get 0xFDFE003814. You don't have to worry about bus numbers because they get taken care of based on the HT chain the to which the packet is routed. This is a memory read/write? Yes. It's just to the reserved addresses for config space. Myles -- coreboot mailing list: coreboot@coreboot.org http://www.coreboot.org/mailman/listinfo/coreboot
Re: [coreboot] unstable AMD Fam10h boot
Hi Patrick, On Tue, Aug 25, 2009 at 02:53:19PM +0200, Patrick Georgi wrote: I have some issues with an Fam10h board here. Every once in a while, boot fails with FIXME! CPU Version unknown or not supported!. The CPU Version mask that is calculated at the location the patch changes is 0x10efff. This value appears if the result of the cpuid call (or NB probe) is 0xff (the highest 8 bits aren't counted in, it might be 0x, too)' I've seen this occasionally, too. I suspected compiler issues again... Thanks, Ward. -- Ward Vandewege w...@fsf.org Free Software Foundation - Senior Systems Administrator -- coreboot mailing list: coreboot@coreboot.org http://www.coreboot.org/mailman/listinfo/coreboot
