Re: IRQ affinity vs. MTRRs, was Re: 36 bit MTRRs, Re: test10-pre1 problems on 4-way SuperServer8050
Boszormenyi Zoltan <[EMAIL PROTECTED]> writes: > The idea is that when it is sure that _only one_ (or some) CPU will access > a PCI card's mmio area then only that CPU's (those CPUs') MTRRs needs to > contain an entry for that area. > > Although there are (must be) common MTRR entries for the main memory > and the commonly accessed mmio register areas. > > The idea came because fiddling with MTRRs quickly revaled that > only 8 variable ones exist. I see. I think there is a more straightforward solution: PAT does the same thing as MTRRs, but has no such "number of ranges" limitation --- it lets you set the memory type on a page-by-page basis. If the number of MTRRs becomes a problem (anyone know how many the P4 has?), then the real solution is to implement PAT support. IIRC, only the PPro, the first PII model (Klamath?), and the first Celeron model have MTRR but not PAT (Athlon has PAT, but /proc/cpuinfo misreports it as "fcmov", at least in 2.2.14; Xeons always had PAT). David - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: IRQ affinity vs. MTRRs, was Re: 36 bit MTRRs, Re: test10-pre1 problems on 4-way SuperServer8050
Boszormenyi Zoltan <[EMAIL PROTECTED]> writes: > I came up with an idea. The MTRRs are per-cpu things. > Ingo Molnar's IRQ affinity code helps binding certain > IRQ sources to certain CPUs. They are implemented as per-cpu things but the Intel manuals say that all cpus should have the same MTRR settings. They also give pseudo-code for how to update them on an SMP system, which mtrr.c follows. If the BIOS has set them up differently at boot time, mtrr.c will complain and copy the MTRR settings of CPU0 to the others. Regards, David Wragg - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: 36 bit MTRRs, Re: test10-pre1 problems on 4-way SuperServer8050
On Thu, 12 Oct 2000, Boszormenyi Zoltan wrote: > On Thu, 12 Oct 2000, Boszormenyi Zoltan wrote: > > > echo "base=0 size=0x1 type=write-back" >/proc/mtrr > > echo "base=0x1 size=0x8000 type=write-back" >/proc/mtrr > > echo "base=0xfe00 size=0x80 type=write-combining" >/proc/mtrr > > echo "base=0xfde0 size=0x10 type=uncached" >/proc/mtrr > > echo "base=0xfe80 size=0x10 type=uncached" >/proc/mtrr > > echo "base=0xfe9ed000 size=0x1000 type=uncached" >/proc/mtrr > > echo "base=0xfe9ee000 size=0x2000 type=uncached" >/proc/mtrr > > echo "base=0xfeafe000 size=0x2000 type=uncached" >/proc/mtrr > > Sorry, use 'uncachable' instead of 'uncached'. :-( ok, doing it from the bottom up was fine (didn't lockup) but reaching the last (first in your list) entry was refused by mtrr: mtrr: 0x0,0x1 overlaps existing 0xfeafe000,0x2000 # cat /proc/mtrr reg00: base=0xfeafe000 (4074MB), size= 0kB: uncachable, count=1 reg01: base=0xfe9ee000 (4073MB), size= 0kB: uncachable, count=1 reg02: base=0xfe9ed000 (4073MB), size= 0kB: uncachable, count=1 reg03: base=0xfe80 (4072MB), size= 1MB: uncachable, count=1 reg04: base=0xfde0 (4062MB), size= 1MB: uncachable, count=1 reg05: base=0xfe00 (4064MB), size= 8MB: write-combining, count=1 reg06: base=0x001 (4096MB), size=2048MB: write-back, count=1 and machine is still slow. So, what is the correct way to cover the 6G by some mtrrs? I will now try to disable or change strategy of L2 caching in BIOS and see if it makes things worse. Regards, Tigran - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: IRQ affinity vs. MTRRs, was Re: 36 bit MTRRs, Re: test10-pre1 problems on 4-way SuperServer8050
On Thu, Oct 12, 2000 at 12:12:19PM +0200, Boszormenyi Zoltan wrote: > I came up with an idea. The MTRRs are per-cpu things. > Ingo Molnar's IRQ affinity code helps binding certain > IRQ sources to certain CPUs. > > What if the MTRR driver allows per-CPU settings, maybe only on > uncached areas? Of course the real memory should be cached in > every CPU to avoid slowdowns. So that if you set that eth0's > IRQ will be handled by CPU1, the MTRRs of CPU1 will be set > accordingly, and the other CPUs will not care about eth0, > so they do not need eth0's MTRR settings. A little question. Why do we want to bind irq of eth0 to a single CPU ? imho it will casue slowdown of some situation. Why don't we leave scheduler to select CPU for processing IRQ ? - Gabor - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: 36 bit MTRRs, Re: test10-pre1 problems on 4-way SuperServer8050
> On Thu, 12 Oct 2000, Boszormenyi Zoltan wrote: > > > echo "base=0 size=0x1 type=write-back" >/proc/mtrr this line immediately locks up the machine. But I want to understand where did you get base=0 and size=0x1 from? Shouldn't it be base=0x10 and size=0xfccf according to this entry from e820: BIOS-e820: fccf @ 0010 (usable) Regards, Tigran - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: 36 bit MTRRs, Re: test10-pre1 problems on 4-way SuperServer8050
On Thu, 12 Oct 2000, Boszormenyi Zoltan wrote: > Look at the e820 map in the boot log, mark those areas > as write-back and tell me what happens. Here is e820 map: BIOS-e820: 0009fc00 @ (usable) BIOS-e820: 0400 @ 0009fc00 (reserved) BIOS-e820: 0002 @ 000e (reserved) BIOS-e820: fccf @ 0010 (usable) BIOS-e820: f000 @ fcdf (ACPI data) BIOS-e820: 1000 @ fcdff000 (ACPI NVS) BIOS-e820: 1000 @ fec0 (reserved) BIOS-e820: 1000 @ fee0 (reserved) BIOS-e820: 0008 @ fff8 (reserved) BIOS-e820: 8000 @ 0001 (usable) I can easily setup the mtrr entry for the top 2G: BIOS-e820: 8000 @ 0001 (usable) # cat /proc/mtrr reg00: base=0x001 (4096MB), size=2048MB: write-combining, count=1 reg02: base=0xfc00 (4032MB), size= 64MB: uncachable, count=1 but trying to do the same for the low 4G: BIOS-e820: fccf @ 0010 (usable) mtrr complains: # echo "base=0x10 size=0xfccf type=write-combining" > /proc/mtrr mtrr: base(0x10) is not aligned on a size(0xfccf) boundary suggestions? Regards, Tigran - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: 36 bit MTRRs, Re: test10-pre1 problems on 4-way SuperServer8050
On Thu, 12 Oct 2000, Boszormenyi Zoltan wrote: Look at the e820 map in the boot log, mark those areas as write-back and tell me what happens. Here is e820 map: BIOS-e820: 0009fc00 @ (usable) BIOS-e820: 0400 @ 0009fc00 (reserved) BIOS-e820: 0002 @ 000e (reserved) BIOS-e820: fccf @ 0010 (usable) BIOS-e820: f000 @ fcdf (ACPI data) BIOS-e820: 1000 @ fcdff000 (ACPI NVS) BIOS-e820: 1000 @ fec0 (reserved) BIOS-e820: 1000 @ fee0 (reserved) BIOS-e820: 0008 @ fff8 (reserved) BIOS-e820: 8000 @ 0001 (usable) I can easily setup the mtrr entry for the top 2G: BIOS-e820: 8000 @ 0001 (usable) # cat /proc/mtrr reg00: base=0x001 (4096MB), size=2048MB: write-combining, count=1 reg02: base=0xfc00 (4032MB), size= 64MB: uncachable, count=1 but trying to do the same for the low 4G: BIOS-e820: fccf @ 0010 (usable) mtrr complains: # echo "base=0x10 size=0xfccf type=write-combining" /proc/mtrr mtrr: base(0x10) is not aligned on a size(0xfccf) boundary suggestions? Regards, Tigran - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: IRQ affinity vs. MTRRs, was Re: 36 bit MTRRs, Re: test10-pre1 problems on 4-way SuperServer8050
On Thu, Oct 12, 2000 at 12:12:19PM +0200, Boszormenyi Zoltan wrote: I came up with an idea. The MTRRs are per-cpu things. Ingo Molnar's IRQ affinity code helps binding certain IRQ sources to certain CPUs. What if the MTRR driver allows per-CPU settings, maybe only on uncached areas? Of course the real memory should be cached in every CPU to avoid slowdowns. So that if you set that eth0's IRQ will be handled by CPU1, the MTRRs of CPU1 will be set accordingly, and the other CPUs will not care about eth0, so they do not need eth0's MTRR settings. A little question. Why do we want to bind irq of eth0 to a single CPU ? imho it will casue slowdown of some situation. Why don't we leave scheduler to select CPU for processing IRQ ? - Gabor - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: 36 bit MTRRs, Re: test10-pre1 problems on 4-way SuperServer8050
On Thu, 12 Oct 2000, Boszormenyi Zoltan wrote: On Thu, 12 Oct 2000, Boszormenyi Zoltan wrote: echo "base=0 size=0x1 type=write-back" /proc/mtrr echo "base=0x1 size=0x8000 type=write-back" /proc/mtrr echo "base=0xfe00 size=0x80 type=write-combining" /proc/mtrr echo "base=0xfde0 size=0x10 type=uncached" /proc/mtrr echo "base=0xfe80 size=0x10 type=uncached" /proc/mtrr echo "base=0xfe9ed000 size=0x1000 type=uncached" /proc/mtrr echo "base=0xfe9ee000 size=0x2000 type=uncached" /proc/mtrr echo "base=0xfeafe000 size=0x2000 type=uncached" /proc/mtrr Sorry, use 'uncachable' instead of 'uncached'. :-( ok, doing it from the bottom up was fine (didn't lockup) but reaching the last (first in your list) entry was refused by mtrr: mtrr: 0x0,0x1 overlaps existing 0xfeafe000,0x2000 # cat /proc/mtrr reg00: base=0xfeafe000 (4074MB), size= 0kB: uncachable, count=1 reg01: base=0xfe9ee000 (4073MB), size= 0kB: uncachable, count=1 reg02: base=0xfe9ed000 (4073MB), size= 0kB: uncachable, count=1 reg03: base=0xfe80 (4072MB), size= 1MB: uncachable, count=1 reg04: base=0xfde0 (4062MB), size= 1MB: uncachable, count=1 reg05: base=0xfe00 (4064MB), size= 8MB: write-combining, count=1 reg06: base=0x001 (4096MB), size=2048MB: write-back, count=1 and machine is still slow. So, what is the correct way to cover the 6G by some mtrrs? I will now try to disable or change strategy of L2 caching in BIOS and see if it makes things worse. Regards, Tigran - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: IRQ affinity vs. MTRRs, was Re: 36 bit MTRRs, Re: test10-pre1 problems on 4-way SuperServer8050
Boszormenyi Zoltan [EMAIL PROTECTED] writes: I came up with an idea. The MTRRs are per-cpu things. Ingo Molnar's IRQ affinity code helps binding certain IRQ sources to certain CPUs. They are implemented as per-cpu things but the Intel manuals say that all cpus should have the same MTRR settings. They also give pseudo-code for how to update them on an SMP system, which mtrr.c follows. If the BIOS has set them up differently at boot time, mtrr.c will complain and copy the MTRR settings of CPU0 to the others. Regards, David Wragg - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: IRQ affinity vs. MTRRs, was Re: 36 bit MTRRs, Re: test10-pre1 problems on 4-way SuperServer8050
Boszormenyi Zoltan [EMAIL PROTECTED] writes: The idea is that when it is sure that _only one_ (or some) CPU will access a PCI card's mmio area then only that CPU's (those CPUs') MTRRs needs to contain an entry for that area. Although there are (must be) common MTRR entries for the main memory and the commonly accessed mmio register areas. The idea came because fiddling with MTRRs quickly revaled that only 8 variable ones exist. I see. I think there is a more straightforward solution: PAT does the same thing as MTRRs, but has no such "number of ranges" limitation --- it lets you set the memory type on a page-by-page basis. If the number of MTRRs becomes a problem (anyone know how many the P4 has?), then the real solution is to implement PAT support. IIRC, only the PPro, the first PII model (Klamath?), and the first Celeron model have MTRR but not PAT (Athlon has PAT, but /proc/cpuinfo misreports it as "fcmov", at least in 2.2.14; Xeons always had PAT). David - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
Re: 36 bit MTRRs, Re: test10-pre1 problems on 4-way SuperServer8050
On Wed, 11 Oct 2000, Boszormenyi Zoltan wrote: > On Wed, 11 Oct 2000, Tigran Aivazian wrote: > > I will continue to narrow down by removing some things (like mtrr) from > > the equation. Rik, the problem is that when one enables PAE (or just > > highmem-4G) support on a 4-way 6G RAM machine becomes 38-40 times slower. > > Will you please try this patch? This is almost the same as I > sent to you before, it is just against 2.4.0-test10-pre1 and > it lacks the corrections to e.g. the frame buffer drivers. Hi Zoltan, I have tried your patch and although it works: # cat /proc/mtrr reg00: base=0x ( 0MB), size=4096MB: write-back, count=1 reg01: base=0x001 (4096MB), size=2048MB: write-back, count=1 reg02: base=0xfc00 (4032MB), size= 64MB: uncachable, count=1 unfortunately, it doesn't solve the problem. The machine is still unbearably slow (up to 40x slower!) and one of the eepro100 interfaces is still not working. Another interesting idea was suggested by Mark Hemment - to switch memlist_add_head() -> memlist_add_tail() in expand()/__free_pages_ok() (see mm/page_alloc.c) and it did make a difference -- both eepro100 started to work fine but the machine remained just as slow as before. So, the problem is complex but I am told that Rik and others are aware that at present there is no working support for highmem. Strangely, my desktop dual PIII550 with 1G RAM works just fine with highmem... nice and fast, no problems whatsoever and it is filled up with all kinds of devices, from soundcard to bttv848, 3dfx, eepro100, ne2k, 8139 etc etc. Everything just works. Regards, Tigran - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
36 bit MTRRs, Re: test10-pre1 problems on 4-way SuperServer8050
On Wed, 11 Oct 2000, Tigran Aivazian wrote: > I will continue to narrow down by removing some things (like mtrr) from > the equation. Rik, the problem is that when one enables PAE (or just > highmem-4G) support on a 4-way 6G RAM machine becomes 38-40 times slower. Will you please try this patch? This is almost the same as I sent to you before, it is just against 2.4.0-test10-pre1 and it lacks the corrections to e.g. the frame buffer drivers. I am now running test10-pre1 with this patch and: - [root@localhost /root]# cd /proc [root@localhost /proc]# cat mtrr reg00: base=0x ( 0MB), size= 128MB: write-back, count=1 reg05: base=0xe200 (3616MB), size= 32MB: write-combining, count=1 [root@localhost /proc]# echo "base=0x2 size=0x1 type=write-combining" >mtrr [root@localhost /proc]# cat mtrr reg00: base=0x ( 0MB), size= 128MB: write-back, count=1 reg01: base=0x002 (8192MB), size=4096MB: write-combining, count=1 reg05: base=0xe200 (3616MB), size= 32MB: write-combining, count=1 -- This is on a dual P-III machine with 128 MB memory. If it causes problems on Athlons then change the line #define AMD_OR_MASK(0xf000UL) to #define AMD_OR_MASK(INTEL_OR_MASK) and recompile and tell me what happens. Also, I would like to hear reports from non-Intel (Cyrix, etc.) and older AMD machines. I do not have my Cyrix 6x86MX anymore but this scheme worked on that. Regards, Zoltan Boszormenyi mtrrpage-new.diff.gz
36 bit MTRRs, Re: test10-pre1 problems on 4-way SuperServer8050
On Wed, 11 Oct 2000, Tigran Aivazian wrote: I will continue to narrow down by removing some things (like mtrr) from the equation. Rik, the problem is that when one enables PAE (or just highmem-4G) support on a 4-way 6G RAM machine becomes 38-40 times slower. Will you please try this patch? This is almost the same as I sent to you before, it is just against 2.4.0-test10-pre1 and it lacks the corrections to e.g. the frame buffer drivers. I am now running test10-pre1 with this patch and: - [root@localhost /root]# cd /proc [root@localhost /proc]# cat mtrr reg00: base=0x ( 0MB), size= 128MB: write-back, count=1 reg05: base=0xe200 (3616MB), size= 32MB: write-combining, count=1 [root@localhost /proc]# echo "base=0x2 size=0x1 type=write-combining" mtrr [root@localhost /proc]# cat mtrr reg00: base=0x ( 0MB), size= 128MB: write-back, count=1 reg01: base=0x002 (8192MB), size=4096MB: write-combining, count=1 reg05: base=0xe200 (3616MB), size= 32MB: write-combining, count=1 -- This is on a dual P-III machine with 128 MB memory. If it causes problems on Athlons then change the line #define AMD_OR_MASK(0xf000UL) to #define AMD_OR_MASK(INTEL_OR_MASK) and recompile and tell me what happens. Also, I would like to hear reports from non-Intel (Cyrix, etc.) and older AMD machines. I do not have my Cyrix 6x86MX anymore but this scheme worked on that. Regards, Zoltan Boszormenyi mtrrpage-new.diff.gz
Re: 36 bit MTRRs, Re: test10-pre1 problems on 4-way SuperServer8050
On Wed, 11 Oct 2000, Boszormenyi Zoltan wrote: On Wed, 11 Oct 2000, Tigran Aivazian wrote: I will continue to narrow down by removing some things (like mtrr) from the equation. Rik, the problem is that when one enables PAE (or just highmem-4G) support on a 4-way 6G RAM machine becomes 38-40 times slower. Will you please try this patch? This is almost the same as I sent to you before, it is just against 2.4.0-test10-pre1 and it lacks the corrections to e.g. the frame buffer drivers. Hi Zoltan, I have tried your patch and although it works: # cat /proc/mtrr reg00: base=0x ( 0MB), size=4096MB: write-back, count=1 reg01: base=0x001 (4096MB), size=2048MB: write-back, count=1 reg02: base=0xfc00 (4032MB), size= 64MB: uncachable, count=1 unfortunately, it doesn't solve the problem. The machine is still unbearably slow (up to 40x slower!) and one of the eepro100 interfaces is still not working. Another interesting idea was suggested by Mark Hemment - to switch memlist_add_head() - memlist_add_tail() in expand()/__free_pages_ok() (see mm/page_alloc.c) and it did make a difference -- both eepro100 started to work fine but the machine remained just as slow as before. So, the problem is complex but I am told that Rik and others are aware that at present there is no working support for highmem. Strangely, my desktop dual PIII550 with 1G RAM works just fine with highmem... nice and fast, no problems whatsoever and it is filled up with all kinds of devices, from soundcard to bttv848, 3dfx, eepro100, ne2k, 8139 etc etc. Everything just works. Regards, Tigran - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [EMAIL PROTECTED] Please read the FAQ at http://www.tux.org/lkml/
