Problems with AMD64 and 8 GB RAM?
I've recently acquired an AMD64 box (dual Opteron 242, SiS [EMAIL PROTECTED] motherboard (http://www.msi.com.tw/program/products/server/svr/pro_svr_detail.php?UID=484). See below for more details). I find it very unstable running with 8 GB memory, though 4 GB are not a problem. At first I thought it was the onboard peripherals, but after disabling them it still persisted. What's unstable? I only once got it through the boot process. Running a 5.3-RELEASE i386 kernel it panics, though I haven't investigated the panic (yet), since I'm not interested in the i386 kernel. The amd64 5.4-PRERELEASE kernel just hangs/freezes. When the peripherals are enabled, it's after probing the onboard NIC (bge) and before probing SATA (no drives present). I've done a verbose boot, of course, but no additional information is present. The NIC is recognized, and that's all. Without the peripherals, but with a 3Com 3c905 PCI NIC, it continues beyond this point, but doesn't enable the NIC. I don't have dmesg output for these attempts, so I can't produce the exact message, and I suspect it's not important. It continues until trying to mount NFS file systems, where it hangs for obvious reasons. Pressing ^C causes the system to either panic (and be unable to dump because I don't have that much swap) or just hang. None of these problems occur when I use 4 GB memory. About the only strangeness, which seems to come from the BIOS, is that it recognizes only 3.5 GB. If I put all DIMMS in, it recognizes the full 8 GB memory. I realize that this isn't enough to diagnose the problem. The reason for this message now is to ask: 1. Has anybody else seen this problem? 2. Has anybody else used this hardware configuration and *not* seen this problem? 3. Where should I look next? I'm attaching the (non-verbose) dmesg from a successful boot. Greg -- See complete headers for address and phone numbers. Mar 30 14:17:16 obelix kernel: FreeBSD 5.4-PRERELEASE #0: Tue Mar 22 04:02:17 UTC 2005 Mar 30 14:17:16 obelix kernel: [EMAIL PROTECTED]:/usr/obj/src/FreeBSD/OBELIX/src/sys/OBELIX Mar 30 14:17:16 obelix kernel: Timecounter "i8254" frequency 1193182 Hz quality 0 Mar 30 14:17:16 obelix kernel: CPU: AMD Opteron(tm) Processor 242 (1603.65-MHz K8-class CPU) Mar 30 14:17:16 obelix kernel: Origin = "AuthenticAMD" Id = 0xf5a Stepping = 10 Mar 30 14:17:16 obelix kernel: Features=0x78bfbff Mar 30 14:17:16 obelix kernel: AMD Features=0xe0500800 Mar 30 14:17:16 obelix kernel: real memory = 3756916736 (3582 MB) Mar 30 14:17:16 obelix kernel: avail memory = 3623907328 (3456 MB) Mar 30 14:17:16 obelix kernel: ACPI APIC Table: Mar 30 14:17:16 obelix kernel: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs Mar 30 14:17:16 obelix kernel: cpu0 (BSP): APIC ID: 0 Mar 30 14:17:16 obelix kernel: cpu1 (AP): APIC ID: 1 Mar 30 14:17:16 obelix kernel: ioapic0: Changing APIC ID to 2 Mar 30 14:17:16 obelix kernel: ioapic0 irqs 0-23 on motherboard Mar 30 14:17:16 obelix kernel: acpi0: on motherboard Mar 30 14:17:16 obelix kernel: acpi0: Power Button (fixed) Mar 30 14:17:16 obelix kernel: Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 Mar 30 14:17:16 obelix kernel: acpi_timer0: <24-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0 Mar 30 14:17:16 obelix kernel: cpu0: on acpi0 Mar 30 14:17:16 obelix kernel: cpu1: on acpi0 Mar 30 14:17:16 obelix kernel: acpi_button0: on acpi0 Mar 30 14:17:16 obelix kernel: pcib0: port 0xcf8-0xcff on acpi0 Mar 30 14:17:16 obelix kernel: pci0: on pcib0 Mar 30 14:17:16 obelix kernel: pcib1: at device 1.0 on pci0 Mar 30 14:17:16 obelix kernel: pci1: on pcib1 Mar 30 14:17:16 obelix kernel: pci1: at device 0.0 (no driver attached) Mar 30 14:17:16 obelix kernel: xl0: <3Com 3c905C-TX Fast Etherlink XL> port 0xd000-0xd07f mem 0xfb00-0xfb7f irq 18 at device 7.0 on pci0 Mar 30 14:17:16 obelix kernel: miibus0: on xl0 Mar 30 14:17:16 obelix kernel: xlphy0: <3c905C 10/100 internal PHY> on miibus0 Mar 30 14:17:16 obelix kernel: xlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto Mar 30 14:17:16 obelix kernel: xl0: Ethernet address: 00:50:da:cf:17:d3 Mar 30 14:17:16 obelix kernel: atapci0: port 0xd400-0xd40f,0x376,0x170-0x177,0x3f6,0x1f0-0 x1f7 at device 15.0 on pci0 Mar 30 14:17:16 obelix kernel: ata0: channel #0 on atapci0 Mar 30 14:17:16 obelix kernel: ata1: channel #1 on atapci0 Mar 30 14:17:16 obelix kernel: uhci0: port 0xd800-0xd81f irq 21 at device 16.0 on pci0 Mar 30 14:17:16 obelix kernel: usb0: on uhci0 Mar 30 14:17:16 obelix kernel: usb0: USB revision 1.0 Mar 30 14:17:16 obelix kernel: uhub0: VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1 Mar 30 14:17:16 obelix kernel: uhub0: 2 ports with 2 removable, self powered Mar 30 14:17:16 obelix kernel: uhci1: port 0xdc00-0xdc1f irq 21 at device 16.1 on pci0 Mar 30 14:17:16 obelix kernel: usb1: on uhci1 Mar 30 14:17:16 obelix kernel: usb1: USB revision 1.0 Mar 30 14:17:16 obelix kernel: uhub1: VIA UHCI root hu
Re: Problems with AMD64 and 8 GB RAM?
Greg 'groggy' Lehey wrote: I've recently acquired an AMD64 box (dual Opteron 242, SiS [EMAIL PROTECTED] motherboard (http://www.msi.com.tw/program/products/server/svr/pro_svr_detail.php?UID=484). See below for more details). I find it very unstable running with 8 GB memory, though 4 GB are not a problem. At first I thought it was the onboard peripherals, but after disabling them it still persisted. What's unstable? I only once got it through the boot process. Running a 5.3-RELEASE i386 kernel it panics, though I haven't investigated the panic (yet), since I'm not interested in the i386 kernel. The amd64 5.4-PRERELEASE kernel just hangs/freezes. When the peripherals are enabled, it's after probing the onboard NIC (bge) and before probing SATA (no drives present). I've done a verbose boot, of course, but no additional information is present. The NIC is recognized, and that's all. Without the peripherals, but with a 3Com 3c905 PCI NIC, it continues beyond this point, but doesn't enable the NIC. I don't have dmesg output for these attempts, so I can't produce the exact message, and I suspect it's not important. It continues until trying to mount NFS file systems, where it hangs for obvious reasons. Pressing ^C causes the system to either panic (and be unable to dump because I don't have that much swap) or just hang. None of these problems occur when I use 4 GB memory. About the only strangeness, which seems to come from the BIOS, is that it recognizes only 3.5 GB. If I put all DIMMS in, it recognizes the full 8 GB memory. I realize that this isn't enough to diagnose the problem. The reason for this message now is to ask: 1. Has anybody else seen this problem? 2. Has anybody else used this hardware configuration and *not* seen this problem? 3. Where should I look next? I'm attaching the (non-verbose) dmesg from a successful boot. Greg -- See complete headers for address and phone numbers. 5.3-RELEASE has a lot of problems with >4GB due to busdma issues. Those should no longer be an issue in RELENG_5, including 5.4-PRE. You'll need to dig in and provide some more details, I guess. I have an HDAMA dual Opteron system that behaves fine now with 8GB of RAM, so your problem might lie with particular hardware and/or drivers. Scott ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM?
On Thu, Mar 31, 2005 at 07:54:39AM +0930, Greg 'groggy' Lehey wrote: > None of these problems occur when I use 4 GB memory. About the only > strangeness, which seems to come from the BIOS, is that it recognizes > only 3.5 GB. If I put all DIMMS in, it recognizes the full 8 GB > memory. > > I realize that this isn't enough to diagnose the problem. The reason > for this message now is to ask: > > 1. Has anybody else seen this problem? > 2. Has anybody else used this hardware configuration and *not* seen > this problem? > 3. Where should I look next? > Have you run sysutils/memtest86 with the 8 GB? I had 4 bad out of 12 tested where the DIMMs were Crucial PC2700 2GB Reg. ECC DIMMs. -- Steve ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM?
On Wednesday, 30 March 2005 at 15:30:37 -0700, Scott Long wrote: > Greg 'groggy' Lehey wrote: >> I've recently acquired an AMD64 box ... >> >> What's unstable? ... The amd64 5.4-PRERELEASE kernel just >> hangs/freezes. > > 5.3-RELEASE has a lot of problems with >4GB due to busdma issues. > Those should no longer be an issue in RELENG_5, including 5.4-PRE. They appear to be. >> I realize that this isn't enough to diagnose the problem. The reason >> for this message now is to ask: >> >> 3. Where should I look next? > > You'll need to dig in and provide some more details, I guess. Yes, my guess too. > I have an HDAMA dual Opteron system that behaves fine now with 8GB > of RAM, so your problem might lie with particular hardware and/or > drivers. As I described, it doesn't appear to be the drivers. Greg -- See complete headers for address and phone numbers. pgpuBz6NIFzNC.pgp Description: PGP signature
Re: Problems with AMD64 and 8 GB RAM?
On Wednesday, 30 March 2005 at 14:35:46 -0800, Steve Kargl wrote: > On Thu, Mar 31, 2005 at 07:54:39AM +0930, Greg 'groggy' Lehey wrote: >> None of these problems occur when I use 4 GB memory. About the only >> strangeness, which seems to come from the BIOS, is that it recognizes >> only 3.5 GB. If I put all DIMMS in, it recognizes the full 8 GB >> memory. >> >> I realize that this isn't enough to diagnose the problem. The reason >> for this message now is to ask: >> >> 1. Has anybody else seen this problem? >> 2. Has anybody else used this hardware configuration and *not* seen >> this problem? >> 3. Where should I look next? > > Have you run sysutils/memtest86 with the 8 GB? Heh. Difficult when the system doesn't run. > I had 4 bad out of 12 tested where the DIMMs were Crucial PC2700 2GB > Reg. ECC DIMMs. OK, this makes sense. It might also explain why the 4 GB configuration only recognizes 3.5 GB. Greg -- See complete headers for address and phone numbers. pgpHUROsJz6Ax.pgp Description: PGP signature
Re: Problems with AMD64 and 8 GB RAM?
On Thursday, 31 March 2005 at 10:32:33 +0930, Daniel O'Connor wrote: > On Thu, 31 Mar 2005 08:14, Greg 'groggy' Lehey wrote: >>> Have you run sysutils/memtest86 with the 8 GB? >> >> Heh. Difficult when the system doesn't run. > > You could try http://www.memtest86.com although that doesn't do >4Gb :( I'm pretty sure it's not the memory. I've tried each pair individually, and it's only when they're both in there together that it's a problem. And yes, I've tried them in each pair of slots. I now have dmesg output for verbose boots with both 4 GB and 8 GB memory. The complete dmesg output is at http://www.lemis.com/grog/Images/20050331/dmesg.4GB and http://www.lemis.com/grog/Images/20050331/dmesg.8GB. The diffs are at http://www.lemis.com/grog/Images/20050331/dmesg.diff. Here's a truncated summary: > --- dmesg.4GB Thu Mar 31 10:47:16 2005 > +++ dmesg.8GB Thu Mar 31 10:52:32 2005 > @@ -64,6 +10,7 @@ > SMAP type=01 base=0010 len=dfde > SMAP type=03 base=dfee3000 len=d000 > SMAP type=04 base=dfee len=3000 > +SMAP type=01 base=0001 len=0001 > Copyright (c) 1992-2005 The FreeBSD Project. > @@ -75,7 +22,7 @@ > Calibrating clock(s) ... i8254 clock: 1193283 Hz > CLK_USE_I8254_CALIBRATION not specified - using default frequency > Timecounter "i8254" frequency 1193182 Hz quality 0 > -Calibrating TSC clock ... TSC clock: 1603647337 Hz > +Calibrating TSC clock ... TSC clock: 1603647241 Hz > CPU: AMD Opteron(tm) Processor 242 (1603.65-MHz K8-class CPU) >Origin = "AuthenticAMD" Id = 0xf5a Stepping = 10 > > Features=0x78bfbff > @@ -90,11 +37,12 @@ > L2 4KB data TLB: 512 entries, 4-way associative > L2 4KB instruction TLB: 512 entries, 4-way associative > L2 unified cache: 1024 kbytes, 64 bytes/line, 1 lines/tag, 16-way associative > -real memory = 3756916736 (3582 MB) > +real memory = 8589934592 (8192 MB) This is interesting in that it has gained 4.5 GB. > Physical memory chunk(s): > 0x1000 - 0x0009bfff, 634880 bytes (155 pages) > -0x00a05000 - 0xd95b7fff, 3636146176 bytes (887731 pages) > -avail memory = 3623817216 (3455 MB) > +0x00a09000 - 0xdfed, 3746394112 bytes (914647 pages) > +0x0001 - 0x0001f0fc, 4043112448 bytes (987088 pages) > +avail memory = 177600 (7416 MB) > ACPI APIC Table: > APIC ID: physical 0, logical 0:0 > APIC ID: physical 1, logical 0:1 > @@ -138,41 +86,12 @@ > ioapic0: intpin 9 trigger: level > ioapic0: intpin 9 polarity: low > lapic0: Routing NMI -> LINT1 > -A IRQ 3 (edge, high) > -ioapic0: intpin 4 -> ISA IRQ 4 (edge, high) > -ioapic0: intpin 5 -> ISA IRQ 5 (edge, high) > -ioapic0: intpin 6 -> ISA IRQ 6 (edge, high) > -ioapic0: intpin 7 -> ISA IRQ 7 (edge, high) > -ioapic0: intpin 8 -> ISA IRQ 8 (edge, high) > -ioapic0: intpin 9 -> ISA IRQ 9 (edge, high) > -ioapic0: intpin 10 -> ISA IRQ 10 (edge, high) > -ioapic0: intpin 11 -> ISA IRQ 11 (edge, high) > -ioapic0: intpin 12 -> ISA IRQ 12 (edge, high) > -ioapic0: intpin 13 -> ISA IRQ 13 (edge, high) > -ioapic0: intpin 14 -> ISA IRQ 14 (edge, high) > -ioapic0: intpin 15 -> ISA IRQ 15 (edge, high) > -ioapic0: intpin 16 -> PCI IRQ 16 (level, low) > -ioapic0: intpin 17 -> PCI IRQ 17 (level, low) > -ioapic0: intpin 18 -> PCI IRQ 18 (level, low) > -ioapic0: intpin 19 -> PCI IRQ 19 (level, low) > -ioapic0: intpin 20 -> PCI IRQ 20 (level, low) > -ioapic0: intpin 21 -> PCI IRQ 21 (level, low) > -ioapic0: intpin 22 -> PCI IRQ 22 (level, low) > -ioapic0: intpin 23 -> PCI IRQ 23 (level, low) > -MADT: Interrupt override: source 0, irq 2 > -ioapic0: Routing IRQ 0 -> intpin 2 > -ioapic0: intpin 2 trigger: edge > -ioapic0: intpin 2 polarity: high > -MADT: Interrupt override: source 9, irq 9 > -ioapic0: intpin 9 trigger: level > -ioapic0: intpin 9 polarity: low > -lapic0: Routing NMI -> LINT1 This stuff is puzzling. I suppose it could be related. Does anybody have any ideas? > lapic0: LINT1 trigger: edge > lapic0: LINT1 polarity: high > lapic1: Routing NMI -> LINT1 > lapic1: LINT1 trigger: edge > lapic1: LINT1 polarity: high > -ioapic0 irqs 0-23 on motherboard > +ioapic0 irqs 0-23 on motherboard > cpu0 BSP: > ID: 0x VER: 0x00040010 LDR: 0x0100 DFR: 0x0fff >lint0: 0x00010700 lint1: 0x0400 TPR: 0x SVR: 0x01ff The last lines in the 8 GB dmesg are: > bge0: mem > 0xfa00-0xfa00 irq 16 at device 11.0 on pci0 > bge0: Reserved 0x1 bytes for rid 0x10 type 3 at 0xfa00 They're identical in each probe. Greg -- See complete headers for address and phone numbers. pgpZU27XHUxhn.pgp Description: PGP signature
Re: Problems with AMD64 and 8 GB RAM?
On Thu, 31 Mar 2005 11:24, Greg 'groggy' Lehey wrote: > I'm pretty sure it's not the memory. I've tried each pair > individually, and it's only when they're both in there together that > it's a problem. And yes, I've tried them in each pair of slots. Could be a marginal timing issue.. You could try winding out the RAM timing slightly. -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C pgpYD49Q7hkAC.pgp Description: PGP signature
Re: Problems with AMD64 and 8 GB RAM?
Greg 'groggy' Lehey wrote: >I'm pretty sure it's not the memory. I've tried each pair >individually, and it's only when they're both in there together that >it's a problem. And yes, I've tried them in each pair of slots. I'm sure you have checked this aswell but just for completeness, they aren't different pairs? Like one pair is single-sided and the other double-sided (had some nasty and obscure problems with such a combination myself)? mkb. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM?
On Thursday, 31 March 2005 at 5:54:17 +0200, Matthias Buelow wrote: > Greg 'groggy' Lehey wrote: > >> I'm pretty sure it's not the memory. I've tried each pair >> individually, and it's only when they're both in there together that >> it's a problem. And yes, I've tried them in each pair of slots. > > I'm sure you have checked this aswell but just for completeness, > they aren't different pairs? Like one pair is single-sided and the > other double-sided (had some nasty and obscure problems with such > a combination myself)? No, they're all the same. Greg -- See complete headers for address and phone numbers. pgpB6Tf3lyzwW.pgp Description: PGP signature
Re: Problems with AMD64 and 8 GB RAM?
On Mar 30, 2005, at 8:54 PM, Greg 'groggy' Lehey wrote: lapic0: LINT1 trigger: edge lapic0: LINT1 polarity: high lapic1: Routing NMI -> LINT1 lapic1: LINT1 trigger: edge lapic1: LINT1 polarity: high -ioapic0 irqs 0-23 on motherboard +ioapic0 irqs 0-23 on motherboard cpu0 BSP: ID: 0x VER: 0x00040010 LDR: 0x0100 DFR: 0x0fff lint0: 0x00010700 lint1: 0x0400 TPR: 0x SVR: 0x01ff This shows that in the - case the APIC is broken somehow (0.0 isn't a valid I/O APIC version). It would seem that the system has mapped RAM over top of the I/O APIC perhaps? It would be interesting to see the contents of your MADT to see if it's trying to use a 64-bit PA for your APIC. The local APIC portion seems ok though. -- John Baldwin <[EMAIL PROTECTED]> <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve" = http://www.FreeBSD.org ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM?
On Wednesday, 30 March 2005 at 23:01:03 -0500, John Baldwin wrote: > > On Mar 30, 2005, at 8:54 PM, Greg 'groggy' Lehey wrote: >>> lapic0: LINT1 trigger: edge >>> lapic0: LINT1 polarity: high >>> lapic1: Routing NMI -> LINT1 >>> lapic1: LINT1 trigger: edge >>> lapic1: LINT1 polarity: high >>> -ioapic0 irqs 0-23 on motherboard >>> +ioapic0 irqs 0-23 on motherboard >>> cpu0 BSP: >>> ID: 0x VER: 0x00040010 LDR: 0x0100 DFR: 0x0fff >>> lint0: 0x00010700 lint1: 0x0400 TPR: 0x SVR: 0x01ff > > This shows that in the - case the APIC is broken somehow (0.0 isn't a > valid I/O APIC version). You mean the + case, I suppose. Yes, that's what I suspected. > It would seem that the system has mapped RAM over top of the I/O > APIC perhaps? That's what I suspected too, but imp doesn't think so. > It would be interesting to see the contents of your MADT to see if > it's trying to use a 64-bit PA for your APIC. Any suggestions about how to do so? Greg -- See complete headers for address and phone numbers. pgpswZz69jMbN.pgp Description: PGP signature
Re: Problems with AMD64 and 8 GB RAM?
Greg 'groggy' Lehey wrote: On Wednesday, 30 March 2005 at 23:01:03 -0500, John Baldwin wrote: On Mar 30, 2005, at 8:54 PM, Greg 'groggy' Lehey wrote: lapic0: LINT1 trigger: edge lapic0: LINT1 polarity: high lapic1: Routing NMI -> LINT1 lapic1: LINT1 trigger: edge lapic1: LINT1 polarity: high -ioapic0 irqs 0-23 on motherboard +ioapic0 irqs 0-23 on motherboard cpu0 BSP: ID: 0x VER: 0x00040010 LDR: 0x0100 DFR: 0x0fff lint0: 0x00010700 lint1: 0x0400 TPR: 0x SVR: 0x01ff This shows that in the - case the APIC is broken somehow (0.0 isn't a valid I/O APIC version). You mean the + case, I suppose. Yes, that's what I suspected. It would seem that the system has mapped RAM over top of the I/O APIC perhaps? That's what I suspected too, but imp doesn't think so. I'd be more inclined to believe that there is an erroneous mapping by the OS, not that things are fundamentally broken in hardware. Your SMAP table shows everything correctly. It's becoming hard to break through your pre-concieved notions here and explain how things actually work. It would be interesting to see the contents of your MADT to see if it's trying to use a 64-bit PA for your APIC. Any suggestions about how to do so? man acpidump ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM?
[gratuitous empty lines removed] On Wednesday, 30 March 2005 at 21:28:36 -0700, Scott Long wrote: > Greg 'groggy' Lehey wrote: >> On Wednesday, 30 March 2005 at 23:01:03 -0500, John Baldwin wrote: >> >>> On Mar 30, 2005, at 8:54 PM, Greg 'groggy' Lehey wrote: >>> > lapic0: LINT1 trigger: edge > lapic0: LINT1 polarity: high > lapic1: Routing NMI -> LINT1 > lapic1: LINT1 trigger: edge > lapic1: LINT1 polarity: high > -ioapic0 irqs 0-23 on motherboard > +ioapic0 irqs 0-23 on motherboard > cpu0 BSP: >ID: 0x VER: 0x00040010 LDR: 0x0100 DFR: 0x0fff > lint0: 0x00010700 lint1: 0x0400 TPR: 0x SVR: 0x01ff >>> >>> This shows that in the - case the APIC is broken somehow (0.0 isn't a >>> valid I/O APIC version). >> >> You mean the + case, I suppose. Yes, that's what I suspected. >> >>> It would seem that the system has mapped RAM over top of the I/O >>> APIC perhaps? >> >> That's what I suspected too, but imp doesn't think so. > > I'd be more inclined to believe that there is an erroneous mapping > by the OS, not that things are fundamentally broken in hardware. Agreed. This has been my favourite hypothesis all along. But isn't that what jhb is saying? > Your SMAP table shows everything correctly. It's becoming hard to > break through your pre-concieved notions here and explain how things > actually work. No, there's nothing to break through. I think you're just having problems 1. expressing yourself, and 2. understanding what I'm saying. I have no preconceived notions. All I can see here is an antagonistic attitude on your part. What's the problem? You'll recall from my first message that I asked for suggestions about how to approach the issue. jhb provided some; you haven't so far. From what you've written, it's unclear whether you disagree with jhb or not. If you do, why? If you don't, what's your point here? >>> It would be interesting to see the contents of your MADT to see if >>> it's trying to use a 64-bit PA for your APIC. >> >> Any suggestions about how to do so? > > man acpidump How do you run that on a system that won't boot? Greg -- See complete headers for address and phone numbers. pgppI6HefiCEz.pgp Description: PGP signature
Re: Problems with AMD64 and 8 GB RAM?
On 03/30/05 23:14, Greg 'groggy' Lehey wrote: On Wednesday, 30 March 2005 at 21:28:36 -0700, Scott Long wrote: Greg 'groggy' Lehey wrote: On Wednesday, 30 March 2005 at 23:01:03 -0500, John Baldwin wrote: On Mar 30, 2005, at 8:54 PM, Greg 'groggy' Lehey wrote: lapic0: LINT1 trigger: edge lapic0: LINT1 polarity: high lapic1: Routing NMI -> LINT1 lapic1: LINT1 trigger: edge lapic1: LINT1 polarity: high -ioapic0 irqs 0-23 on motherboard +ioapic0 irqs 0-23 on motherboard cpu0 BSP: ID: 0x VER: 0x00040010 LDR: 0x0100 DFR: 0x0fff lint0: 0x00010700 lint1: 0x0400 TPR: 0x SVR: 0x01ff This shows that in the - case the APIC is broken somehow (0.0 isn't a valid I/O APIC version). You mean the + case, I suppose. Yes, that's what I suspected. It would seem that the system has mapped RAM over top of the I/O APIC perhaps? That's what I suspected too, but imp doesn't think so. I'd be more inclined to believe that there is an erroneous mapping by the OS, not that things are fundamentally broken in hardware. Agreed. This has been my favourite hypothesis all along. But isn't that what jhb is saying? Your SMAP table shows everything correctly. It's becoming hard to break through your pre-concieved notions here and explain how things actually work. No, there's nothing to break through. I think you're just having problems 1. expressing yourself, and 2. understanding what I'm saying. I have no preconceived notions. All I can see here is an antagonistic attitude on your part. What's the problem? You'll recall from my first message that I asked for suggestions about how to approach the issue. jhb provided some; you haven't so far. From what you've written, it's unclear whether you disagree with jhb or not. If you do, why? If you don't, what's your point here? It would be interesting to see the contents of your MADT to see if it's trying to use a 64-bit PA for your APIC. Any suggestions about how to do so? man acpidump How do you run that on a system that won't boot? You said the system worked with 4 GB (albeit detecting only 3.5 GB). My perception of this whole ACPI thing is that it is fixed in your BIOS (although it can be overridden by the OS). As such, the amount of RAM you have in the machine shouldn't change acpidump results. Is that not correct? Jon ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM?
Jon Noack wrote: On 03/30/05 23:14, Greg 'groggy' Lehey wrote: On Wednesday, 30 March 2005 at 21:28:36 -0700, Scott Long wrote: Greg 'groggy' Lehey wrote: On Wednesday, 30 March 2005 at 23:01:03 -0500, John Baldwin wrote: On Mar 30, 2005, at 8:54 PM, Greg 'groggy' Lehey wrote: lapic0: LINT1 trigger: edge lapic0: LINT1 polarity: high lapic1: Routing NMI -> LINT1 lapic1: LINT1 trigger: edge lapic1: LINT1 polarity: high -ioapic0 irqs 0-23 on motherboard +ioapic0 irqs 0-23 on motherboard cpu0 BSP: ID: 0x VER: 0x00040010 LDR: 0x0100 DFR: 0x0fff lint0: 0x00010700 lint1: 0x0400 TPR: 0x SVR: 0x01ff This shows that in the - case the APIC is broken somehow (0.0 isn't a valid I/O APIC version). You mean the + case, I suppose. Yes, that's what I suspected. It would seem that the system has mapped RAM over top of the I/O APIC perhaps? That's what I suspected too, but imp doesn't think so. I'd be more inclined to believe that there is an erroneous mapping by the OS, not that things are fundamentally broken in hardware. Agreed. This has been my favourite hypothesis all along. But isn't that what jhb is saying? Your SMAP table shows everything correctly. It's becoming hard to break through your pre-concieved notions here and explain how things actually work. No, there's nothing to break through. I think you're just having problems 1. expressing yourself, and 2. understanding what I'm saying. I have no preconceived notions. All I can see here is an antagonistic attitude on your part. What's the problem? You'll recall from my first message that I asked for suggestions about how to approach the issue. jhb provided some; you haven't so far. From what you've written, it's unclear whether you disagree with jhb or not. If you do, why? If you don't, what's your point here? It would be interesting to see the contents of your MADT to see if it's trying to use a 64-bit PA for your APIC. Any suggestions about how to do so? man acpidump How do you run that on a system that won't boot? You said the system worked with 4 GB (albeit detecting only 3.5 GB). My perception of this whole ACPI thing is that it is fixed in your BIOS (although it can be overridden by the OS). As such, the amount of RAM you have in the machine shouldn't change acpidump results. Is that not correct? Jon This is absolutely correct. Scott ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM?
On Wednesday, 30 March 2005 at 22:27:43 -0700, Scott Long wrote:
> Jon Noack wrote:
>> On 03/30/05 23:14, Greg 'groggy' Lehey wrote:
>>> On Wednesday, 30 March 2005 at 21:28:36 -0700, Scott Long wrote:
Greg 'groggy' Lehey wrote:
> On Wednesday, 30 March 2005 at 23:01:03 -0500, John Baldwin wrote:
>> It would be interesting to see the contents of your MADT to see if
>> it's trying to use a 64-bit PA for your APIC.
>
> Any suggestions about how to do so?
man acpidump
>>>
>>> How do you run that on a system that won't boot?
>>
>> You said the system worked with 4 GB (albeit detecting only 3.5
>> GB).
Yes, this is correct. A number of people have explained why it only
detected 3.5 GB in this configuration.
>> My perception of this whole ACPI thing is that it is fixed in your
>> BIOS (although it can be overridden by the OS). As such, the
>> amount of RAM you have in the machine shouldn't change acpidump
>> results. Is that not correct?
>
> This is absolutely correct.
Ah, so you meant to say that the output from the system running with 4
GB memory is useful? That wasn't in the man page you pointed to.
What it does say is:
> When invoked with the -t flag, the acpidump utility dumps contents of
> the following tables:
>
> ... MADT
This may be the case, but between man page and output some terminology
must have changed. I can't see any reference to anything like an MADT
there. Does that mean that there isn't one, or that ACPI can't find
it, or does the section APIC refer to/dump the MADT? Here's the
complete output of acpidump -t, anyway:
/*
RSD PTR: OEM=VIAK8, ACPI_Rev=1.0x (0)
RSDT=0xdfee3000, cksum=97
*/
/*
RSDT: Length=44, Revision=1, Checksum=4,
OEMID=VIAK8, OEM Table ID=AWRDACPI, OEM Revision=0x42302e31,
Creator ID=AWRD, Creator Revision=0x0
Entries={ 0xdfee3040, 0xdfee7b40 }
*/
/*
FACP: Length=116, Revision=1, Checksum=255,
OEMID=VIAK8, OEM Table ID=AWRDACPI, OEM Revision=0x42302e31,
Creator ID=AWRD, Creator Revision=0x0
FACS=0xdfee, DSDT=0xdfee30c0
INT_MODEL=PIC
Preferred_PM_Profile=Unspecified (0)
SCI_INT=9
SMI_CMD=0x402f, ACPI_ENABLE=0xa1, ACPI_DISABLE=0xa0, S4BIOS_REQ=0x0
PSTATE_CNT=0x0
PM1a_EVT_BLK=0x4000-0x4003
PM1a_CNT_BLK=0x4004-0x4005
PM_TMR_BLK=0x4008-0x400b
GPE0_BLK=0x4020-0x4023
P_LVL2_LAT=101 us, P_LVL3_LAT=1001 us
FLUSH_SIZE=0, FLUSH_STRIDE=0
DUTY_OFFSET=0, DUTY_WIDTH=1
DAY_ALRM=125, MON_ALRM=126, CENTURY=50
IAPC_BOOT_ARCH=
Flags={WBINVD,PROC_C1,SLP_BUTTON,RTC_S4,RESET_REG}
RESET_REG=0x:0[0] (Memory), RESET_VALUE=0x44
*/
/*
FACS: Length=64, HwSig=0x, Firm_Wake_Vec=0x
Global_Lock=
Flags=
Version=0
*/
/*
DSDT: Length=19020, Revision=1, Checksum=28,
OEMID=VIAK8, OEM Table ID=AWRDACPI, OEM Revision=0x1000,
Creator ID=MSFT, Creator Revision=0x10e
*/
/*
APIC: Length=104, Revision=1, Checksum=145,
OEMID=VIAK8, OEM Table ID=AWRDACPI, OEM Revision=0x42302e31,
Creator ID=AWRD, Creator Revision=0x0
Local APIC ADDR=0xfee0
Flags={PC-AT}
Type=Local APIC
ACPI CPU=0
Flags={ENABLED}
APIC ID=0
Type=Local APIC
ACPI CPU=1
Flags={ENABLED}
APIC ID=1
Type=IO APIC
APIC ID=2
INT BASE=0
ADDR=0xfec0
Type=INT Override
BUS=0
IRQ=0
INTR=2
Flags={Polarity=conforming, Trigger=conforming}
Type=INT Override
BUS=0
IRQ=9
INTR=9
Flags={Polarity=active-lo, Trigger=level}
Type=Local NMI
ACPI CPU=0
LINT Pin=1
Flags={Polarity=active-hi, Trigger=edge}
Type=Local NMI
ACPI CPU=1
LINT Pin=1
Flags={Polarity=active-hi, Trigger=edge}
*/
Since I don't know anything about ACPI, this doesn't say too much to
me. Suggestions welcome. If the APIC section is the MADT, it looks
as if we should update the docco.
Greg
--
See complete headers for address and phone numbers.
pgpd4k8dxjan9.pgp
Description: PGP signature
Re: Problems with AMD64 and 8 GB RAM?
On 03/30/05 23:49, Greg 'groggy' Lehey wrote: On Wednesday, 30 March 2005 at 22:27:43 -0700, Scott Long wrote: Jon Noack wrote: On 03/30/05 23:14, Greg 'groggy' Lehey wrote: On Wednesday, 30 March 2005 at 21:28:36 -0700, Scott Long wrote: Greg 'groggy' Lehey wrote: On Wednesday, 30 March 2005 at 23:01:03 -0500, John Baldwin wrote: It would be interesting to see the contents of your MADT to see if it's trying to use a 64-bit PA for your APIC. Any suggestions about how to do so? man acpidump How do you run that on a system that won't boot? You said the system worked with 4 GB (albeit detecting only 3.5 GB). Yes, this is correct. A number of people have explained why it only detected 3.5 GB in this configuration. My perception of this whole ACPI thing is that it is fixed in your BIOS (although it can be overridden by the OS). As such, the amount of RAM you have in the machine shouldn't change acpidump results. Is that not correct? This is absolutely correct. Ah, so you meant to say that the output from the system running with 4 GB memory is useful? That wasn't in the man page you pointed to. What it does say is: When invoked with the -t flag, the acpidump utility dumps contents of the following tables: ... MADT This may be the case, but between man page and output some terminology must have changed. I can't see any reference to anything like an MADT there. Does that mean that there isn't one, or that ACPI can't find it, or does the section APIC refer to/dump the MADT? Here's the complete output of acpidump -t, anyway: Since I don't know anything about ACPI, this doesn't say too much to me. Suggestions welcome. If the APIC section is the MADT, it looks as if we should update the docco. My limited research (as in, Google) shows that the MADT was defined as part of ACPI 2.0: http://www.microsoft.com/whdc/system/platform/64bit/IA64_ACPI.mspx According to your previous link the motherboard specs, it supports both ACPI 1.0A and 2.0. Perhaps there is a BIOS knob to toggle between the two? Jon ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM?
On Thursday, 31 March 2005 at 0:00:22 -0600, Jon Noack wrote: > On 03/30/05 23:49, Greg 'groggy' Lehey wrote: >> Here's the complete output of acpidump -t, anyway: >> >> >> >> Since I don't know anything about ACPI, this doesn't say too much to >> me. Suggestions welcome. If the APIC section is the MADT, it looks >> as if we should update the docco. > > My limited research (as in, Google) shows that the MADT was defined as > part of ACPI 2.0: > http://www.microsoft.com/whdc/system/platform/64bit/IA64_ACPI.mspx Thanks for the link. > According to your previous link the motherboard specs, it supports > both ACPI 1.0A and 2.0. Perhaps there is a BIOS knob to toggle > between the two? I've taken a look, but I can't find anything. Greg -- See complete headers for address and phone numbers. pgpF5N94haxLL.pgp Description: PGP signature
Re: Problems with AMD64 and 8 GB RAM?
Hi, Since we are discussing AMD64 with 8GB RAM, I also would like to point my problem. I'm still looking for possibility to run FreeBSD 5.3-STABLE with more than 4GB RAM on Dual amd64 2.2GHz machine (IBM @server 325) with ServeRAID 6M (ips driver)). Right now I'm using only 4GB RAM and this server is in production. #uname -an FreeBSD publica.ub.mng.net 5.3-STABLE FreeBSD 5.3-STABLE #12: Mon Nov 22 12:04:57 ULAT 2004 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/AMD amd64 As Scott said a few months ago, problem is below: "The ips driver looks like it will fail under heavy load when more than 4GB of RAM is present. It tries to force busdma to not defer requests when the bounce page reserve is low, but that looks to be broken and will result in corrupted commands." Are the ips driver and bus_dma problems fixed yet in STABLE tree? Is it worth to try source update and see how it works? I'm afraid to do so, since it is production server. Please see my previous posts: http://lists.freebsd.org/mailman/htdig/freebsd-current/2004-December/044325.html http://lists.freebsd.org/pipermail/freebsd-current/2004-October/041003.html http://lists.freebsd.org/pipermail/freebsd-current/2004-October/041005.html http://lists.freebsd.org/pipermail/freebsd-current/2004-October/041013.html http://lists.freebsd.org/pipermail/freebsd-current/2004-October/041015.html http://lists.freebsd.org/pipermail/freebsd-current/2004-October/041094.html http://lists.freebsd.org/pipermail/freebsd-current/2004-October/041112.html http://lists.freebsd.org/pipermail/freebsd-current/2004-October/041164.html http://lists.freebsd.org/pipermail/freebsd-current/2004-October/041258.html http://lists.freebsd.org/pipermail/freebsd-current/2004-October/041554.html dmesg: http://lists.freebsd.org/pipermail/freebsd-current/2004-October/041265.html thanks in advance, Ganbold ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM?
Ganbold wrote: Hi, Since we are discussing AMD64 with 8GB RAM, I also would like to point my problem. I'm still looking for possibility to run FreeBSD 5.3-STABLE with more than 4GB RAM on Dual amd64 2.2GHz machine (IBM @server 325) with ServeRAID 6M (ips driver)). Right now I'm using only 4GB RAM and this server is in production. #uname -an FreeBSD publica.ub.mng.net 5.3-STABLE FreeBSD 5.3-STABLE #12: Mon Nov 22 12:04:57 ULAT 2004 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/AMD amd64 As Scott said a few months ago, problem is below: "The ips driver looks like it will fail under heavy load when more than 4GB of RAM is present. It tries to force busdma to not defer requests when the bounce page reserve is low, but that looks to be broken and will result in corrupted commands." Are the ips driver and bus_dma problems fixed yet in STABLE tree? Is it worth to try source update and see how it works? I'm afraid to do so, since it is production server. Yes, I (hopefully) fixed the problems that I pointed out, and I also locked it and added crashdump support. It is reported to be stable and fast now, so you won't go wrong by updating. These changes are in both 5-stable and 6-current. Scott ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM?
On Wednesday 30 March 2005 09:49 pm, Greg 'groggy' Lehey wrote: > On Wednesday, 30 March 2005 at 22:27:43 -0700, Scott Long wrote: > > Jon Noack wrote: > >> On 03/30/05 23:14, Greg 'groggy' Lehey wrote: > >>> On Wednesday, 30 March 2005 at 21:28:36 -0700, Scott Long wrote: > Greg 'groggy' Lehey wrote: > > On Wednesday, 30 March 2005 at 23:01:03 -0500, John Baldwin wrote: > >> It would be interesting to see the contents of your MADT to see if > >> it's trying to use a 64-bit PA for your APIC. > > > > Any suggestions about how to do so? > > man acpidump > >>> > >>> How do you run that on a system that won't boot? > >> > >> You said the system worked with 4 GB (albeit detecting only 3.5 > >> GB). > > Yes, this is correct. A number of people have explained why it only > detected 3.5 GB in this configuration. > You're also being confused by the implementation of the 'real memory' report. If you take a 30 second glance at the code, you'll see that it is reporting the same units that the hw.maxmem tunable uses. ie: it is the LIMIT or Highest Address that the system has, not the sum total of all the parts. eg: see the machdep.c comment next to the printf * Maxmem isn't the "maximum memory", it's one larger than the * highest page of the physical address space. It should be * called something like "Maxphyspage". We may adjust this * based on ``hw.physmem'' and the results of the memory test. The SMAP lines are what you need to pay attention to. In the output you posted with 8G, you can see the 4GB going from the 4->8GB range, exactly. SMAP type 1 is "usable memory". -Peter ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM?
On Mar 31, 2005, at 1:00 AM, Jon Noack wrote: My limited research (as in, Google) shows that the MADT was defined as part of ACPI 2.0: http://www.microsoft.com/whdc/system/platform/64bit/IA64_ACPI.mspx According to your previous link the motherboard specs, it supports both ACPI 1.0A and 2.0. Perhaps there is a BIOS knob to toggle between the two? It's part if ACPI 1.0 as well. Trust me, I have machines built before ACPI 2.0 was defined that have MADTs. :) -- John Baldwin <[EMAIL PROTECTED]> <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve" = http://www.FreeBSD.org ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM?
On Mar 30, 2005, at 11:08 PM, Greg 'groggy' Lehey wrote: On Wednesday, 30 March 2005 at 23:01:03 -0500, John Baldwin wrote: On Mar 30, 2005, at 8:54 PM, Greg 'groggy' Lehey wrote: lapic0: LINT1 trigger: edge lapic0: LINT1 polarity: high lapic1: Routing NMI -> LINT1 lapic1: LINT1 trigger: edge lapic1: LINT1 polarity: high -ioapic0 irqs 0-23 on motherboard +ioapic0 irqs 0-23 on motherboard cpu0 BSP: ID: 0x VER: 0x00040010 LDR: 0x0100 DFR: 0x0fff lint0: 0x00010700 lint1: 0x0400 TPR: 0x SVR: 0x01ff This shows that in the - case the APIC is broken somehow (0.0 isn't a valid I/O APIC version). You mean the + case, I suppose. Yes, that's what I suspected. It would seem that the system has mapped RAM over top of the I/O APIC perhaps? That's what I suspected too, but imp doesn't think so. Actually, if the full version register were zero, it would not have had 24 IRQs (irqs 0-23 part), so I'm not sure what it is doing. 0.3 isn't really a valid APIC version AFAIK either, though I'm more familiar with the versions used in Intel APICs (usually 1.1, 1.2, or 2.0). It would be interesting to see the contents of your MADT to see if it's trying to use a 64-bit PA for your APIC. Any suggestions about how to do so? Boot with 4g or boot an i386 version and get acpidump -t output. -- John Baldwin <[EMAIL PROTECTED]> <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve" = http://www.FreeBSD.org ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM?
On Mar 31, 2005, at 12:49 AM, Greg 'groggy' Lehey wrote:
This may be the case, but between man page and output some terminology
must have changed. I can't see any reference to anything like an MADT
there. Does that mean that there isn't one, or that ACPI can't find
it, or does the section APIC refer to/dump the MADT? Here's the
complete output of acpidump -t, anyway:
MADT is the name of the table (Multiple APIC Descriptor Table or some
such), but "APIC" is the 4 character signature of the MADT, hence
seeing 'APIC' output from acpidump -t when looking at the MADT.
Similarly, the MP Table is known as the MP Table, but the signature for
the table that you search for in the BIOS is "_MP_".
/*
APIC: Length=104, Revision=1, Checksum=145,
OEMID=VIAK8, OEM Table ID=AWRDACPI, OEM Revision=0x42302e31,
Creator ID=AWRD, Creator Revision=0x0
Local APIC ADDR=0xfee0
Flags={PC-AT}
Type=Local APIC
ACPI CPU=0
Flags={ENABLED}
APIC ID=0
Type=Local APIC
ACPI CPU=1
Flags={ENABLED}
APIC ID=1
Type=IO APIC
APIC ID=2
INT BASE=0
ADDR=0xfec0
Type=INT Override
BUS=0
IRQ=0
INTR=2
Flags={Polarity=conforming, Trigger=conforming}
Type=INT Override
BUS=0
IRQ=9
INTR=9
Flags={Polarity=active-lo, Trigger=level}
Type=Local NMI
ACPI CPU=0
LINT Pin=1
Flags={Polarity=active-hi, Trigger=edge}
Type=Local NMI
ACPI CPU=1
LINT Pin=1
Flags={Polarity=active-hi, Trigger=edge}
*/
Nothing strange here, and it is giving a 64-bit PA for the I/O APIC,
albeit one that is < 4GB. One thing to verify is that the physical
addresses listed here for the APICs (0xfec0 and 0xfee0) aren't
included in the SMAP as valid RAM addresses in both cases. It might be
useful to boot an i386 CD with 8GB in the machine to see if the MADT
looks any different in that case.
--
John Baldwin <[EMAIL PROTECTED]> <>< http://www.FreeBSD.org/~jhb/
"Power Users Use the Power to Serve" = http://www.FreeBSD.org
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM?
On Mar 31, 2005, at 12:27 AM, Scott Long wrote: Jon Noack wrote: On 03/30/05 23:14, Greg 'groggy' Lehey wrote: On Wednesday, 30 March 2005 at 21:28:36 -0700, Scott Long wrote: Greg 'groggy' Lehey wrote: On Wednesday, 30 March 2005 at 23:01:03 -0500, John Baldwin wrote: On Mar 30, 2005, at 8:54 PM, Greg 'groggy' Lehey wrote: lapic0: LINT1 trigger: edge lapic0: LINT1 polarity: high lapic1: Routing NMI -> LINT1 lapic1: LINT1 trigger: edge lapic1: LINT1 polarity: high -ioapic0 irqs 0-23 on motherboard +ioapic0 irqs 0-23 on motherboard cpu0 BSP: ID: 0x VER: 0x00040010 LDR: 0x0100 DFR: 0x0fff lint0: 0x00010700 lint1: 0x0400 TPR: 0x SVR: 0x01ff This shows that in the - case the APIC is broken somehow (0.0 isn't a valid I/O APIC version). You mean the + case, I suppose. Yes, that's what I suspected. It would seem that the system has mapped RAM over top of the I/O APIC perhaps? That's what I suspected too, but imp doesn't think so. I'd be more inclined to believe that there is an erroneous mapping by the OS, not that things are fundamentally broken in hardware. Agreed. This has been my favourite hypothesis all along. But isn't that what jhb is saying? Your SMAP table shows everything correctly. It's becoming hard to break through your pre-concieved notions here and explain how things actually work. No, there's nothing to break through. I think you're just having problems 1. expressing yourself, and 2. understanding what I'm saying. I have no preconceived notions. All I can see here is an antagonistic attitude on your part. What's the problem? You'll recall from my first message that I asked for suggestions about how to approach the issue. jhb provided some; you haven't so far. From what you've written, it's unclear whether you disagree with jhb or not. If you do, why? If you don't, what's your point here? It would be interesting to see the contents of your MADT to see if it's trying to use a 64-bit PA for your APIC. Any suggestions about how to do so? man acpidump How do you run that on a system that won't boot? You said the system worked with 4 GB (albeit detecting only 3.5 GB). My perception of this whole ACPI thing is that it is fixed in your BIOS (although it can be overridden by the OS). As such, the amount of RAM you have in the machine shouldn't change acpidump results. Is that not correct? Jon This is absolutely correct. It might though. Notice the change in APIC version with 4GB of RAM vs 8GB. The APIC hardware is the same, so that's already indicative of something fishy going on. I think that his APIC address is correct though as otherwise no interrupts at all would work and it wouldn't claim to have 24 IRQs on the APIC in both cases. One can always boot an i386 non-PAE kernel with 8GB in the machine and get an acpidump though. -- John Baldwin <[EMAIL PROTECTED]> <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve" = http://www.FreeBSD.org ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM?
On Wed, Mar 30, 2005 at 03:25:37PM -0800, Peter Wemm wrote: > Greg: The busdma problems from 5.3-RELEASE are fixed. That doesn't > mean that there are no *other* problems. Scott is saying "the old > busdma bug shouldn't be affecting 5.4-PRE", and he's correct. > > Most likely, something else is happening, eg: you're running out of KVM > or something silly like that. I know we're right on the brink at 8GB. > The layout of the devices may be just enough to tip it over the edge. Grog's motherboard is a 4+0 configuration -- which would mean he is using (trying to) 2GB DIMM's. There are memory bus loading specifictions he may be out of spec of. -- -- David ([EMAIL PROTECTED]) ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM?
On Thu, Mar 31, 2005 at 08:14:45AM +0930, Greg 'groggy' Lehey wrote: > > I had 4 bad out of 12 tested where the DIMMs were Crucial PC2700 2GB > > Reg. ECC DIMMs. > > OK, this makes sense. It might also explain why the 4 GB > configuration only recognizes 3.5 GB. No. This is due to the 3.5-4.0GB PA address range that the PeeCee architecture reserves for the PCI config space, AGP GART, memory mapped I/O, etc... Many Opteron BIOS's don't bother to hoist the "covered" memory above 4GB. Please see the freebsd-amd64 archives -- this has been discussed many times. -- -- David ([EMAIL PROTECTED]) ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM?
On Thu, Mar 31, 2005 at 10:32:33AM +0930, Daniel O'Connor wrote: > On Thu, 31 Mar 2005 08:14, Greg 'groggy' Lehey wrote: > > > Have you run sysutils/memtest86 with the 8 GB? > > > > Heh. Difficult when the system doesn't run. > > You could try http://www.memtest86.com although that doesn't do >4Gb :( Are you sure version 3.2 does not? -- -- David ([EMAIL PROTECTED]) ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM?
On Thu, Mar 31, 2005 at 11:24:29AM +0930, Greg 'groggy' Lehey wrote: > On Thursday, 31 March 2005 at 10:32:33 +0930, Daniel O'Connor wrote: > > On Thu, 31 Mar 2005 08:14, Greg 'groggy' Lehey wrote: > >>> Have you run sysutils/memtest86 with the 8 GB? > >> > >> Heh. Difficult when the system doesn't run. > > > > You could try http://www.memtest86.com although that doesn't do >4Gb > :( > > I'm pretty sure it's not the memory. I've tried each pair > individually, and it's only when they're both in there together that > it's a problem. And yes, I've tried them in each pair of slots. You have a dual-channel memory controller. If you insert one DIMM you perform 64-bit data accesses. If you install DIMM's in pairs (making sure you're using the right "paired" sockets), you perform 128-bit data accesses. Thus your access pattern is different between these two situations. I'm highly suspious that you can us 4x2GB DIMM's with out knowing the exact part number. Don't forget 2GB DIMM's are double-stacked and thus look like double the electrical bus loads. The same is true for older 1GB DIMM's. Install all the memory you would like to use into your motherboard, download memtest86+ version 1.40 from http://www.memtest.org, dd to floppy or burn the ISO, and report back your findings from running it. Also what version of the BIOS are you using? -- -- David ([EMAIL PROTECTED]) ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM?
> Date: Thu, 31 Mar 2005 16:12:35 +0900 > From: Ganbold <[EMAIL PROTECTED]> > Subject: Re: Problems with AMD64 and 8 GB RAM? > > Hi, > > Since we are discussing AMD64 with 8GB RAM, I also would like to point my > problem. > > I'm still looking for possibility to run FreeBSD 5.3-STABLE with more than > 4GB RAM > on Dual amd64 2.2GHz machine (IBM @server 325) with ServeRAID 6M (ips > driver)). > Right now I'm using only 4GB RAM and this server is in production. > > #uname -an > FreeBSD publica.ub.mng.net 5.3-STABLE FreeBSD 5.3-STABLE #12: Mon Nov 22 > 12:04:57 ULAT 2004 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/AMD > amd64 > > As Scott said a few months ago, problem is below: > > "The ips driver looks like it will fail under heavy load when more than 4GB > of RAM is present. It tries to force busdma to not defer requests when the > bounce page reserve is low, but that looks to be broken and > will result in corrupted commands." [Alan Jay] Since we are talking about FreeBSD on AMD64 on the AMD64 list I have reported issues on that list. I have a TyanThunder K8S pro S2882 twin Operteron with 8Gb of RAM and although I can get the machine to run reasonably stably with 8Gb of RAM with limited loading when pushed it falls over unpredictably. We did some tests with the latest 5.3-STABLE / 5.4-PRERELEASE and still found the same issues when using a mySQL database heavily hit over the Ethernet controller. Our final tests limited the memory on boot-up to 4Gb and the bug is still there so we think it may well be some interaction with the Ethernet controller. The motherboard we have has a BroadcomBCM5704C 10/100/1000 based card on board. Again this works fine initially but then we get a very dramatic failure with no warning messages and the system falls over. There are still a few issues to be ironed out with the FreeBSD 5.x on AMD64 the latest STABLE/PRE-RELEASE is much improved but be aware there may be issues. We will be waiting a few more weeks before re-trying these tests to see if the latest fixes that have been discussed have solved our problems. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM?
On Mar 31, 2005, at 3:08 PM, Alan Jay wrote: We did some tests with the latest 5.3-STABLE / 5.4-PRERELEASE and still found the same issues when using a mySQL database heavily hit over the Ethernet controller. Our final tests limited the memory on boot-up to 4Gb and the bug is still there so we think it may well be some interaction with the Ethernet controller. The motherboard we have has a BroadcomBCM5704C 10/100/1000 based card on board. I have seen similar with Postgres 8.0 database. Occasionally I'll see a bge0 timout + reset error logged, but many times I'll just see a "socket closed unexpectedly" type of message from postgres. So far, every 5 days or so, the machine freezes during heavy DB reporting over the net. I have a S2881 mobo, though, with 4GB. I had another identical machine which was reporting in the BIOS that the memory size changed during normal operations, which was very scary... Vivek Khera, Ph.D. +1-301-869-4449 x806 ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM?
Greg 'groggy' Lehey wrote: I've recently acquired an AMD64 box (dual Opteron 242, SiS [EMAIL PROTECTED] motherboard (http://www.msi.com.tw/program/products/server/svr/pro_svr_detail.php?UID=484). See below for more details). I find it very unstable running with 8 GB memory, though 4 GB are not a problem. At first I thought it was the onboard peripherals, but after disabling them it still persisted. What's unstable? I only once got it through the boot process. Running a 5.3-RELEASE i386 kernel it panics, though I haven't investigated the panic (yet), since I'm not interested in the i386 kernel. The amd64 5.4-PRERELEASE kernel just hangs/freezes. When the peripherals are enabled, it's after probing the onboard NIC (bge) and before probing SATA (no drives present). I've done a verbose boot, of course, but no additional information is present. The NIC is recognized, and that's all. Without the peripherals, but with a 3Com 3c905 PCI NIC, it continues beyond this point, but doesn't enable the NIC. I don't have dmesg output for these attempts, so I can't produce the exact message, and I suspect it's not important. It continues until trying to mount NFS file systems, where it hangs for obvious reasons. Pressing ^C causes the system to either panic (and be unable to dump because I don't have that much swap) or just hang. None of these problems occur when I use 4 GB memory. About the only strangeness, which seems to come from the BIOS, is that it recognizes only 3.5 GB. If I put all DIMMS in, it recognizes the full 8 GB memory. I realize that this isn't enough to diagnose the problem. The reason for this message now is to ask: 1. Has anybody else seen this problem? Hi Greg, [Currently little time so I'll dig the archives later for more details] I'm sorry to come into this discussion after 58 messages, but this board has been extensively discussed about 1 year ago, because it gave me trouble to no end (even with 2Gb). One of the early amd64 developers (not David or Scott) had the same board but could not get it stable under amd64 (i386 was fine with 2Gb). He tossed it, and suggested me to do the same. Which I did, and went to a Tyan board S278. After that there where no more problems at all. At the time I think things we're at 5.1 so now with 5.3 some features might have made the board act more stable. --WjW ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM?
Greg 'groggy' Lehey wrote: I've recently acquired an AMD64 box (dual Opteron 242, SiS [EMAIL PROTECTED] motherboard (http://www.msi.com.tw/program/products/server/svr/pro_svr_detail.php?UID=484). See below for more details). I find it very unstable running with 8 GB memory, though 4 GB are not a problem. At first I thought it was the onboard peripherals, but after disabling them it still persisted. What's unstable? I only once got it through the boot process. Running a 5.3-RELEASE i386 kernel it panics, though I haven't 1. Has anybody else seen this problem? 2. Has anybody else used this hardware configuration and *not* seen this problem? [Posted something like thisearlier, but did not see it on the list] Little late to the discussion, but none the less. I bought this board over a year ago to run amd64 (you're running i386). But in the end I trashed it for running amd64 since one of then involved developers also tried to use the board without much success. It was running fine with amd64, 1 CPU, 2Gb, but as soon as I added the 2nd CPU the slightest load crashed the system somewhere in IPI-areas. After long discussions I came to the point that it was easier to get a new motherboard, so I got a Tyan Tiger S275. Which as not yet failed on me. So if ever you'd like to run amd64 on this system, even now you've determined the problem to be the load on the memory-bus, be warned that odd things could be happening. --WjW ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM?
On Tue, Apr 05, 2005 at 09:33:05AM +0200, Willem Jan Withagen wrote: > I'm sorry to come into this discussion after 58 messages, but this board > has been extensively discussed about 1 year ago, because it gave me trouble > to no end (even with 2Gb). One of the early amd64 developers (not David or > Scott) had the same board but could not get it stable under amd64 (i386 was > fine with 2Gb). He tossed it, and suggested me to do the same. Hogwash. It was Peter Wemm and he was talking about the Asus SK8N, not the MSI K8T Master2-FAR. -- -- David ([EMAIL PROTECTED]) ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM?
David O'Brien wrote: On Tue, Apr 05, 2005 at 09:33:05AM +0200, Willem Jan Withagen wrote: I'm sorry to come into this discussion after 58 messages, but this board has been extensively discussed about 1 year ago, because it gave me trouble to no end (even with 2Gb). One of the early amd64 developers (not David or Scott) had the same board but could not get it stable under amd64 (i386 was fine with 2Gb). He tossed it, and suggested me to do the same. Hogwash. It was Peter Wemm and he was talking about the Asus SK8N, not the MSI K8T Master2-FAR. Not true unfortunatley I cannot seem to find the old mails on this. Probably got lost when upgrading my WinBox. But I'm very shure about this, why else would I burn a nice board and get me an expensive new one??? It is still somewhere in my storage I had several people look at my kerneldumps. (I even needed to fix a small bug for that in doadump().) Nobody seemed to be able to really explain what was wrong, and everything magically went away when I bought the Tyan board. So please don't tell me I was smoking something illegal. --WjW ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM?
On Thu, Mar 31, 2005 at 08:14:45AM +0930, Greg 'groggy' Lehey wrote: > On Wednesday, 30 March 2005 at 14:35:46 -0800, Steve Kargl wrote: > > On Thu, Mar 31, 2005 at 07:54:39AM +0930, Greg 'groggy' Lehey wrote: > >> None of these problems occur when I use 4 GB memory. About the only > >> strangeness, which seems to come from the BIOS, is that it recognizes > >> only 3.5 GB. If I put all DIMMS in, it recognizes the full 8 GB > >> memory. > >> > >> I realize that this isn't enough to diagnose the problem. The reason > >> for this message now is to ask: > >> > >> 3. Where should I look next? > > > > Have you run sysutils/memtest86 with the 8 GB? > > Heh. Difficult when the system doesn't run. That's what happens when 1 of 8 (1 of 4?) DIMM is bad :-) > > I had 4 bad out of 12 tested where the DIMMs were Crucial PC2700 2GB > > Reg. ECC DIMMs. > > OK, this makes sense. It might also explain why the 4 GB > configuration only recognizes 3.5 GB. Search amd64 mailing list. The missing memory is reserved for something which escapes me at the moment. Similar to the infamous ISA memory hole. -- Steve ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM?
Greg 'groggy' Lehey wrote: On Wednesday, 30 March 2005 at 14:35:46 -0800, Steve Kargl wrote: On Thu, Mar 31, 2005 at 07:54:39AM +0930, Greg 'groggy' Lehey wrote: None of these problems occur when I use 4 GB memory. About the only strangeness, which seems to come from the BIOS, is that it recognizes only 3.5 GB. If I put all DIMMS in, it recognizes the full 8 GB memory. I realize that this isn't enough to diagnose the problem. The reason for this message now is to ask: 1. Has anybody else seen this problem? 2. Has anybody else used this hardware configuration and *not* seen this problem? 3. Where should I look next? Have you run sysutils/memtest86 with the 8 GB? Heh. Difficult when the system doesn't run. I had 4 bad out of 12 tested where the DIMMs were Crucial PC2700 2GB Reg. ECC DIMMs. OK, this makes sense. It might also explain why the 4 GB configuration only recognizes 3.5 GB. No, and I'm going to make this an FAQ and post it in a very obvious place, since 4+ GB is so easy to get and people don't seem to understand the PC architecture very well. Almost all systems put the PCI Memory Mapped IO window into the 3.75-4GB region of the physical memory map. The registers for the APICs and other system resources are also typically in this region. Now with PCI-Express, the Memory Mapped PCI config registers are typically being mapped in the 3.5-3.75GB range. The memory controllers, host bridges, north-bridges, and/or whatever else glues the memory to the bus to the CPU decode these addresses into PCI cycles, not RAM cycles. Some systems are smart and re-map the RAM that is hidden by these holes into a region >4GB. Some systems are dumb, though, and just deny you access to the RAM that is covered up. It's very much like the old days of the XT/AT architecture when you had 1MB of RAM but everything above 640k was hidden by the VGA framebuffer, ISA option ROMs, and system BIOS, but some systems where smart enough to relocate the hidden RAM. So, your missing .5GB is almost certainly not due to defective RAM, it's just due to The Way Things Are. It's a lot harder for Opteron systems to be smart about this than Xeon systems since all of the remapping magic can happen in the hostbridge on the Xeon, while the Opertons need to have their built-in memory controllers programmed specially for it. Scott ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM?
Greg 'groggy' Lehey wrote: On Wednesday, 30 March 2005 at 15:30:37 -0700, Scott Long wrote: Greg 'groggy' Lehey wrote: I've recently acquired an AMD64 box ... What's unstable? ... The amd64 5.4-PRERELEASE kernel just hangs/freezes. 5.3-RELEASE has a lot of problems with >4GB due to busdma issues. Those should no longer be an issue in RELENG_5, including 5.4-PRE. They appear to be. I don't understand what you mean here. I realize that this isn't enough to diagnose the problem. The reason for this message now is to ask: 3. Where should I look next? You'll need to dig in and provide some more details, I guess. Yes, my guess too. I have an HDAMA dual Opteron system that behaves fine now with 8GB of RAM, so your problem might lie with particular hardware and/or drivers. As I described, it doesn't appear to be the drivers. I don't see how you proved or disproved this. Scott ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM?
On Wednesday, 30 March 2005 at 16:04:44 -0700, Scott Long wrote: > Greg 'groggy' Lehey wrote: >> On Wednesday, 30 March 2005 at 15:30:37 -0700, Scott Long wrote: >> >>> Greg 'groggy' Lehey wrote: >>> I've recently acquired an AMD64 box ... What's unstable? ... The amd64 5.4-PRERELEASE kernel just hangs/freezes. >>> >>> 5.3-RELEASE has a lot of problems with >4GB due to busdma issues. >>> Those should no longer be an issue in RELENG_5, including 5.4-PRE. >> >> They appear to be. > > I don't understand what you mean here. As I said above (and trimmed for convenience), this problem occurs on 5.4-PRERELEASE as of yesterday morning. The dmesg shows that too. >> As I described, it doesn't appear to be the drivers. > > I don't see how you proved or disproved this. Shall I resend the original message? It seems independent of any particular driver. That's not proof, of course, but I didn't claim it was. Greg -- See complete headers for address and phone numbers. pgppvFZmjt3W8.pgp Description: PGP signature
Re: Problems with AMD64 and 8 GB RAM?
On Wednesday, 30 March 2005 at 16:01:14 -0700, Scott Long wrote: > Greg 'groggy' Lehey wrote: >> On Wednesday, 30 March 2005 at 14:35:46 -0800, Steve Kargl wrote: >> >>> On Thu, Mar 31, 2005 at 07:54:39AM +0930, Greg 'groggy' Lehey wrote: >>> None of these problems occur when I use 4 GB memory. About the only strangeness, which seems to come from the BIOS, is that it recognizes only 3.5 GB. If I put all DIMMS in, it recognizes the full 8 GB memory. >>> >>> I had 4 bad out of 12 tested where the DIMMs were Crucial PC2700 2GB >>> Reg. ECC DIMMs. >> >> OK, this makes sense. It might also explain why the 4 GB >> configuration only recognizes 3.5 GB. > > No, and I'm going to make this an FAQ and post it in a very obvious > place, since 4+ GB is so easy to get and people don't seem to understand > the PC architecture very well. That's not easy to understand when it's barely documented. Thanks for the info: it helps a lot. This may still be a hint, though: that memory hole doesn't show up during a boot with 8 GB RAM. How come? Is the system trying to map RAM over the PCI hole? It looks as if I should get a verbose boot listing with 8 GB. It'll be a couple of hours before I find time to reboot this machine. In the meantime, there's a verbose boot with 4 GB at http://www.lemis.com/grog/Images/20050331/obelix-dmesg. I'm told it shows a number of strange things, including incorrect reporting of on-chip cache sizes. Greg -- See complete headers for address and phone numbers. pgpqVG3rp0uyU.pgp Description: PGP signature
Re: Problems with AMD64 and 8 GB RAM?
On Wednesday, 30 March 2005 at 14:57:15 -0800, Steve Kargl wrote: > On Thu, Mar 31, 2005 at 08:14:45AM +0930, Greg 'groggy' Lehey wrote: >> On Wednesday, 30 March 2005 at 14:35:46 -0800, Steve Kargl wrote: >>> On Thu, Mar 31, 2005 at 07:54:39AM +0930, Greg 'groggy' Lehey wrote: None of these problems occur when I use 4 GB memory. About the only strangeness, which seems to come from the BIOS, is that it recognizes only 3.5 GB. If I put all DIMMS in, it recognizes the full 8 GB memory. I realize that this isn't enough to diagnose the problem. The reason for this message now is to ask: 3. Where should I look next? >>> >>> Have you run sysutils/memtest86 with the 8 GB? >> >> Heh. Difficult when the system doesn't run. > > That's what happens when 1 of 8 (1 of 4?) DIMM is bad :-) I've booted with the other 2 DIMMs now (I have 4 2 GB DIMMs, all the MB will hold). No problems. See my last reply to Scott: I'm wondering if the system is ignoring the PCI hole. Greg -- See complete headers for address and phone numbers. pgpoGoHsF272S.pgp Description: PGP signature
Re: Problems with AMD64 and 8 GB RAM?
On Wednesday 30 March 2005 03:15 pm, Greg 'groggy' Lehey wrote: > On Wednesday, 30 March 2005 at 16:01:14 -0700, Scott Long wrote: > > Greg 'groggy' Lehey wrote: > >> On Wednesday, 30 March 2005 at 14:35:46 -0800, Steve Kargl wrote: > >>> On Thu, Mar 31, 2005 at 07:54:39AM +0930, Greg 'groggy' Lehey wrote: > None of these problems occur when I use 4 GB memory. About the > only strangeness, which seems to come from the BIOS, is that it > recognizes only 3.5 GB. If I put all DIMMS in, it recognizes > the full 8 GB memory. > >>> > >>> I had 4 bad out of 12 tested where the DIMMs were Crucial PC2700 > >>> 2GB Reg. ECC DIMMs. > >> > >> OK, this makes sense. It might also explain why the 4 GB > >> configuration only recognizes 3.5 GB. > > > > No, and I'm going to make this an FAQ and post it in a very obvious > > place, since 4+ GB is so easy to get and people don't seem to > > understand the PC architecture very well. > > That's not easy to understand when it's barely documented. Thanks > for the info: it helps a lot. > > This may still be a hint, though: that memory hole doesn't show up > during a boot with 8 GB RAM. How come? Is the system trying to map > RAM over the PCI hole? Nope, its still there. When you boot -v, you'll see the hole in the "Physical memory chunk(s)" list. However, I suspect that some of the bioses will set the 4GB hole partition in the physical ram lower so that there will be 4.5GB of ram above the 4GB mark. I haven't looked too closely to see for sure. > It looks as if I should get a verbose boot listing with 8 GB. It'll > be a couple of hours before I find time to reboot this machine. In > the meantime, there's a verbose boot with 4 GB at > http://www.lemis.com/grog/Images/20050331/obelix-dmesg. I'm told it > shows a number of strange things, including incorrect reporting of > on-chip cache sizes. Nope, it is correct. You have 1MB of L2 cache. L1 data cache: 64 kbytes, 64 bytes/line, 1 lines/tag, 2-way associative L1 instruction cache: 64 kbytes, 64 bytes/line, 1 lines/tag, 2-way associative L2 unified cache: 1024 kbytes, 64 bytes/line, 1 lines/tag, 16-way associative > Greg > -- > See complete headers for address and phone numbers. -- Peter Wemm - [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] "All of this is for nothing if we don't go to the stars" - JMS/B5 ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM?
.. Original Message ... On Thu, 31 Mar 2005 08:14:45 +0930 "Greg 'groggy' Lehey" <[EMAIL PROTECTED]> wrote: >> Have you run sysutils/memtest86 with the 8 GB? > >Heh. Difficult when the system doesn't run. There is a bootable ISO version of memtest86 that you could try. - ask -- http://askask.com/ ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM?
Greg 'groggy' Lehey wrote: On Wednesday, 30 March 2005 at 16:01:14 -0700, Scott Long wrote: Greg 'groggy' Lehey wrote: On Wednesday, 30 March 2005 at 14:35:46 -0800, Steve Kargl wrote: On Thu, Mar 31, 2005 at 07:54:39AM +0930, Greg 'groggy' Lehey wrote: None of these problems occur when I use 4 GB memory. About the only strangeness, which seems to come from the BIOS, is that it recognizes only 3.5 GB. If I put all DIMMS in, it recognizes the full 8 GB memory. I had 4 bad out of 12 tested where the DIMMs were Crucial PC2700 2GB Reg. ECC DIMMs. OK, this makes sense. It might also explain why the 4 GB configuration only recognizes 3.5 GB. No, and I'm going to make this an FAQ and post it in a very obvious place, since 4+ GB is so easy to get and people don't seem to understand the PC architecture very well. That's not easy to understand when it's barely documented. Thanks for the info: it helps a lot. This may still be a hint, though: that memory hole doesn't show up during a boot with 8 GB RAM. How come? Is the system trying to map RAM over the PCI hole? It looks as if I should get a verbose boot listing with 8 GB. It'll be a couple of hours before I find time to reboot this machine. In the meantime, there's a verbose boot with 4 GB at http://www.lemis.com/grog/Images/20050331/obelix-dmesg. I'm told it shows a number of strange things, including incorrect reporting of on-chip cache sizes. The SMAP will show the hole. It's well documented in most PC archtitecure books. Scott ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM?
Greg 'groggy' Lehey wrote: On Wednesday, 30 March 2005 at 16:04:44 -0700, Scott Long wrote: Greg 'groggy' Lehey wrote: On Wednesday, 30 March 2005 at 15:30:37 -0700, Scott Long wrote: Greg 'groggy' Lehey wrote: I've recently acquired an AMD64 box ... What's unstable? ... The amd64 5.4-PRERELEASE kernel just hangs/freezes. 5.3-RELEASE has a lot of problems with >4GB due to busdma issues. Those should no longer be an issue in RELENG_5, including 5.4-PRE. They appear to be. I don't understand what you mean here. As I said above (and trimmed for convenience), this problem occurs on 5.4-PRERELEASE as of yesterday morning. The dmesg shows that too. And you're certain that it's due to the same busdma issues that I was describing? I must have missed the evidence that you use to support this. As I described, it doesn't appear to be the drivers. I don't see how you proved or disproved this. Shall I resend the original message? It seems independent of any particular driver. That's not proof, of course, but I didn't claim it was. Again, I must have missed the part where you investigated the drivers that apply to your particular system. I highly doubt that they apply to every 8GB Opteron system available on the market. Scott ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM?
On Wednesday 30 March 2005 03:09 pm, Greg 'groggy' Lehey wrote: > On Wednesday, 30 March 2005 at 16:04:44 -0700, Scott Long wrote: > > Greg 'groggy' Lehey wrote: > >> On Wednesday, 30 March 2005 at 15:30:37 -0700, Scott Long wrote: > >>> Greg 'groggy' Lehey wrote: > I've recently acquired an AMD64 box ... > > What's unstable? ... The amd64 5.4-PRERELEASE kernel just > hangs/freezes. > >>> > >>> 5.3-RELEASE has a lot of problems with >4GB due to busdma issues. > >>> Those should no longer be an issue in RELENG_5, including > >>> 5.4-PRE. > >> > >> They appear to be. > > > > I don't understand what you mean here. > > As I said above (and trimmed for convenience), this problem occurs on > 5.4-PRERELEASE as of yesterday morning. The dmesg shows that too. > > >> As I described, it doesn't appear to be the drivers. > > > > I don't see how you proved or disproved this. > > Shall I resend the original message? It seems independent of any > particular driver. That's not proof, of course, but I didn't claim > it was. Greg: The busdma problems from 5.3-RELEASE are fixed. That doesn't mean that there are no *other* problems. Scott is saying "the old busdma bug shouldn't be affecting 5.4-PRE", and he's correct. Most likely, something else is happening, eg: you're running out of KVM or something silly like that. I know we're right on the brink at 8GB. The layout of the devices may be just enough to tip it over the edge. -- Peter Wemm - [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] "All of this is for nothing if we don't go to the stars" - JMS/B5 ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM?
On Wednesday 30 March 2005 03:22 pm, Ask Bjørn Hansen wrote: > .. Original Message ... > On Thu, 31 Mar 2005 08:14:45 +0930 "Greg 'groggy' Lehey" > <[EMAIL PROTECTED]> > > wrote: > >> Have you run sysutils/memtest86 with the 8 GB? > > > >Heh. Difficult when the system doesn't run. > > There is a bootable ISO version of memtest86 that you could try. Thats what the port does.. It produces a bootable floppy or ISO. -- Peter Wemm - [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] "All of this is for nothing if we don't go to the stars" - JMS/B5 ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM?
In message: <[EMAIL PROTECTED]> "Greg 'groggy' Lehey" <[EMAIL PROTECTED]> writes: : I've booted with the other 2 DIMMs now (I have 4 2 GB DIMMs, all the : MB will hold). No problems. See my last reply to Scott: I'm : wondering if the system is ignoring the PCI hole. Unlikely. If it was, you'd not have enough of a system to complain about. Warner ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM?
[Format recovered--see http://www.lemis.com/email/email-format.html] [gratuitous empty lines removed] On Wednesday, 30 March 2005 at 16:23:34 -0700, Scott Long wrote: > Greg 'groggy' Lehey wrote: >> On Wednesday, 30 March 2005 at 16:04:44 -0700, Scott Long wrote: >>> Greg 'groggy' Lehey wrote: On Wednesday, 30 March 2005 at 15:30:37 -0700, Scott Long wrote: > Greg 'groggy' Lehey wrote: >> I've recently acquired an AMD64 box ... >> >> What's unstable? ... The amd64 5.4-PRERELEASE kernel just >> hangs/freezes. > > 5.3-RELEASE has a lot of problems with >4GB due to busdma issues. > Those should no longer be an issue in RELENG_5, including 5.4-PRE. They appear to be. >>> >>> I don't understand what you mean here. >> >> As I said above (and trimmed for convenience), this problem occurs on >> 5.4-PRERELEASE as of yesterday morning. The dmesg shows that too. > > And you're certain that it's due to the same busdma issues that I > was describing? No. > I must have missed the evidence that you use to support this. I didn't give any. It appears that I misunderstood what you were saying. As I described, it doesn't appear to be the drivers. >>> >>> I don't see how you proved or disproved this. >> >> Shall I resend the original message? It seems independent of any >> particular driver. That's not proof, of course, but I didn't claim it >> was. > > Again, I must have missed the part where you investigated the drivers > that apply to your particular system. The description is still there. > I highly doubt that they apply to every 8GB Opteron system available > on the market. I never suggested that they did. There's every reason to believe that it's something to do with this particular motherboard, but that doesn't mean that FreeBSD is blameless. Greg -- When replying to this message, please take care not to mutilate the original text. For more information, see http://www.lemis.com/email.html See complete headers for address and phone numbers. pgp0VFy61yFYS.pgp Description: PGP signature
Re: Problems with AMD64 and 8 GB RAM?
On Wednesday, 30 March 2005 at 15:25:37 -0800, Peter Wemm wrote: > On Wednesday 30 March 2005 03:09 pm, Greg 'groggy' Lehey wrote: >> On Wednesday, 30 March 2005 at 16:04:44 -0700, Scott Long wrote: >>> Greg 'groggy' Lehey wrote: As I described, it doesn't appear to be the drivers. >>> >>> I don't see how you proved or disproved this. >> >> Shall I resend the original message? It seems independent of any >> particular driver. That's not proof, of course, but I didn't claim >> it was. > > Greg: The busdma problems from 5.3-RELEASE are fixed. That doesn't > mean that there are no *other* problems. Scott is saying "the old > busdma bug shouldn't be affecting 5.4-PRE", and he's correct. Yes, now I understand. > Most likely, something else is happening, eg: you're running out of KVM > or something silly like that. I know we're right on the brink at 8GB. > The layout of the devices may be just enough to tip it over the edge. Yes, this seems reasonable. Where should I look next? I'm currently rebuilding world and will attempt a verbose boot via serial console when it's done. Anything else I should try? Greg -- See complete headers for address and phone numbers. pgpV3oeVrvAB0.pgp Description: PGP signature
Re: Problems with AMD64 and 8 GB RAM?
On Thu, 31 Mar 2005 08:14, Greg 'groggy' Lehey wrote: > > Have you run sysutils/memtest86 with the 8 GB? > > Heh. Difficult when the system doesn't run. You could try http://www.memtest86.com although that doesn't do >4Gb :( -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C pgpXvfkMAM1yn.pgp Description: PGP signature
Re: Problems with AMD64 and 8 GB RAM?
On Thu, Mar 31, 2005 at 10:32:33AM +0930, Daniel O'Connor wrote: > On Thu, 31 Mar 2005 08:14, Greg 'groggy' Lehey wrote: > > > Have you run sysutils/memtest86 with the 8 GB? > > > > Heh. Difficult when the system doesn't run. > > You could try http://www.memtest86.com although that doesn't do >4Gb :( > http://www.memtest.org/ -- Steve ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM?
On Thu, 31 Mar 2005 10:40, Steve Kargl wrote: > On Thu, Mar 31, 2005 at 10:32:33AM +0930, Daniel O'Connor wrote: > > On Thu, 31 Mar 2005 08:14, Greg 'groggy' Lehey wrote: > > > > Have you run sysutils/memtest86 with the 8 GB? > > > > > > Heh. Difficult when the system doesn't run. > > > > You could try http://www.memtest86.com although that doesn't do >4Gb :( > > http://www.memtest.org/ Ahh well there you go :) Thanks! -- Daniel O'Connor software and network engineer for Genesis Software - http://www.gsoft.com.au "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C pgp8NYdVZHABS.pgp Description: PGP signature
Re: Problems with AMD64 and 8 GB RAM? (solved)
On Thursday, 31 March 2005 at 10:59:02 -0800, David O'Brien wrote: > On Thu, Mar 31, 2005 at 11:24:29AM +0930, Greg 'groggy' Lehey wrote: >> On Thursday, 31 March 2005 at 10:32:33 +0930, Daniel O'Connor wrote: >>> On Thu, 31 Mar 2005 08:14, Greg 'groggy' Lehey wrote: > Have you run sysutils/memtest86 with the 8 GB? Heh. Difficult when the system doesn't run. >>> >>> You could try http://www.memtest86.com although that doesn't do >4Gb >> :( >> >> I'm pretty sure it's not the memory. I've tried each pair >> individually, and it's only when they're both in there together that >> it's a problem. And yes, I've tried them in each pair of slots. > > You have a dual-channel memory controller. If you insert one DIMM you > perform 64-bit data accesses. If you install DIMM's in pairs (making > sure you're using the right "paired" sockets), you perform 128-bit data > accesses. Thus your access pattern is different between these two > situations. I'm highly suspious that you can us 4x2GB DIMM's with out > knowing the exact part number. Don't forget 2GB DIMM's are > double-stacked and thus look like double the electrical bus loads. The > same is true for older 1GB DIMM's. This looks like it's the issue. > Install all the memory you would like to use into your motherboard, > download memtest86+ version 1.40 from http://www.memtest.org, dd to > floppy or burn the ISO, and report back your findings from running > it. Done that. It was quite revealing. This particular motherboard and BIOS supply either "ECC off" or "ECC in chip kill mode". Mine was off, and I got many errors. With 4 GB (any two chips) and chip kill mode enabled, I ran memtest86+ for about 18 hours and got about 1 ECC error per second. With 8 GB, with or without chip kill, neither FreeBSD, memtest86+ nor Linux run reliably: memtest86+ spontaneously reboots every 5 minutes or so. I borrowed this memory to test the motherboard, and it's going back this week anyway. I now have my own memory :-(all 1 GB of it), and it tests perfectly. The memory I had the trouble with works perfectly in the machine for which it was purchased, so it does indeed look like an electrical loading problem. The moral of the story is, I suppose, "don't buy the MSI K8T Master2-FAR". I was warned about the motherboard before I bought it, but at the time it was the only game in town. Now I know of other places with (hopefully) better boards. Greg -- See complete headers for address and phone numbers. pgpTvRjcpU4Il.pgp Description: PGP signature
Re: Problems with AMD64 and 8 GB RAM? (solved)
This may not be related at all, but I've been following this thread and at certain times I wanted to drop in and stress that things like pci express cards should be re-inserted after the RAM has been (re)inserted. Perhaps the BIOSes are really that stupid that they only rely on memory adresses assigned by when you plugged in what. Anyway, it may not matter at all for you but knowing the kind of inquiring mind you have, and considering that this was one of the "sorta-would-it-matter" things hanging over this discussion, perhaps it's a point of interest also. This is how I understand it: FreeBSD 5.4-PRERELEASE #0: Fri Apr 1 01:16:43 CEST 2005 root@:/usr/obj/usr/src/sys/GENERIC Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Pentium(R) 4 CPU 3.40GHz (3400.12-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0xf34 Stepping = 4 Features=0xbfebfbff Hyperthreading: 2 logical CPUs real memory = 1071837184 (1022 MB) avail memory = 1035120640 (987 MB) mptable_probe: MP Config Table has bad signature: \M^?\M^?\M^?\M^? ACPI APIC Table: [part of dmesg with this board] I have a rather new uniprocessor Intel board and the manual specifically stressed (in the light of the 4 BG limit and the &*^*&^:-o things BIOSes might do to overcome that -- the bios will want to have that reserved adress pool just below the 4 GB and depending on the bios it can map it above 4GB perhaps) that any pci express cards (my graphics card) must be inserted AFTER the RAM. Also, about the pairing of DDR2 RAM: you must alight the ram in both channels in principle (that is both in A not B), but if you don't it works if you have the same totals. Again, mobo docs should have all the quirky rules. I didn't want to disturb the discussion but in retrospective, I feel that perhaps some of these points might matter. My observations are on i386 not amd64 but the same kind of logic seems to apply AFAICT. HTH, (in retrospective of course ;-) Dan ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM? (solved)
On Tue, Apr 05, 2005 at 10:09:11AM +0930, Greg 'groggy' Lehey wrote: > The moral of the story is, I suppose, "don't buy the MSI K8T > Master2-FAR". I was warned about the motherboard before I bought it, WHY?? There is nothing wrong with that motherboard -- I have three of them. You are at fault for trying to treat a consumer desktop 2P mobo as a high-end server one. People do not run 2GB DIMM's in an MSI K8T Master2-FAR. If you want >4GB RAM get an eATX 4+4 DIMM configuration motherboard from Tyan or Iwill. The MSI K8T Master2-FAR works fine with 1GB DIMM's. Where on MSI's website did they state 2GB double-stacked DIMM's were supported? -- -- David ([EMAIL PROTECTED]) ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM? (solved)
On Fri, 2005-04-08 at 13:04 -0700, David O'Brien wrote: > The MSI K8T Master2-FAR works fine with 1GB DIMM's. Where on MSI's > website did they state 2GB double-stacked DIMM's were supported? I haven't been following the discussion, but if we're talking about the MSI K8T Master2-FAR, one can read the following from MSI's website: " • Supports DIMM sizes from 64MB (128Mb x 16 DRAMs) to 2GB. • Supports 4 DDR DIMMs up to 8GB (Registered Memory only) " http://www.msi.com.tw/program/products/server/svr/pro_svr_detail.php?UID=484 :-) -- Joel ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Problems with AMD64 and 8 GB RAM? (solved)
On Friday, 8 April 2005 at 13:04:18 -0700, David O'Brien wrote: > On Tue, Apr 05, 2005 at 10:09:11AM +0930, Greg 'groggy' Lehey wrote: >> The moral of the story is, I suppose, "don't buy the MSI K8T >> Master2-FAR". I was warned about the motherboard before I bought it, > > WHY?? There is nothing wrong with that motherboard -- I have three of > them. You are at fault for trying to treat a consumer desktop 2P mobo as > a high-end server one. "You are at fault for trying to used a mobo in an advertised configuration". Sorry, David. If a board doesn't perform as advertised, it's inferior. If you can find a way to work around, fine. I can too, and I have done. But the ability to find a workaround for a deficiency is no reason to say "There is nothing wrong with that motherboard". > People do not run 2GB DIMM's in an MSI K8T Master2-FAR. Indeed. You can't. > The MSI K8T Master2-FAR works fine with 1GB DIMM's. Where on MSI's > website did they state 2GB double-stacked DIMM's were supported? I can't be bothered to search their web site. It's on page 2-8 of the instructions supplied with the motherboard (document G52-S9130X3, version 1.2). Greg -- See complete headers for address and phone numbers. pgpOjAHZUNWP1.pgp Description: PGP signature
[OT] memtest86 (Was: Problems with AMD64 and 8 GB RAM?)
On Mar 30, "Peter Wemm" wrote: > On Wednesday 30 March 2005 03:22 pm, Ask Bjørn Hansen wrote: > > .. Original Message ... > > On Thu, 31 Mar 2005 08:14:45 +0930 "Greg 'groggy' Lehey" > > <[EMAIL PROTECTED]> > > > > wrote: > > >> Have you run sysutils/memtest86 with the 8 GB? > > > > > >Heh. Difficult when the system doesn't run. > > > > There is a bootable ISO version of memtest86 that you could try. > > Thats what the port does.. It produces a bootable floppy or ISO. This reminds me, I noticed that gentoo includes a memtest86 "kernel" in their install ISO. Would this be a hard feature to include in FreeBSD? Mike ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[EMAIL PROTECTED]"
