Re: coherence-problem on the mapped memory buffer
on 29/07/2010 17:13 Alexander Fiveg said the following: P.S. Details about hardware and used software: 1. /var/run/dmesg.boot : ... CPU: Dual Core AMD Opteron(tm) Processor 865 (1800.01-MHz 686-class CPU) Origin = AuthenticAMD Id = 0x20f10 Family = f Model = 21 Stepping = 0 Features=0x178bfbffFPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT Features2=0x1SSE3 AMD Features=0xe2500800SYSCALL,NX,MMX+,FFXSR,LM,3DNow!+,3DNow! AMD Features2=0x3LAHF,CMP real memory = 3758030848 (3583 MB) avail memory = 3677495296 (3507 MB) ACPI APIC Table: A M I OEMAPIC FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs FreeBSD/SMP: 4 package(s) x 2 core(s) ... 2. uname -v FreeBSD 9.0-CURRENT #3 3. sysctl kern.osreldate kern.osreldate: 900014 4. //depot/projects/soc2010/ringmap/ No help, but just curious - do use amd64 variant? If yes, can you reproduce the problem with i386? -- Andriy Gapon ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: coherence-problem on the mapped memory buffer
On Thursday 29 July 2010 18:13:23 Andriy Gapon wrote: on 29/07/2010 17:13 Alexander Fiveg said the following: P.S. Details about hardware and used software: 1. /var/run/dmesg.boot : ... CPU: Dual Core AMD Opteron(tm) Processor 865 (1800.01-MHz 686-class CPU) Origin = AuthenticAMD Id = 0x20f10 Family = f Model = 21 Stepping = 0 Features=0x178bfbffFPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE, MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT Features2=0x1SSE3 AMD Features=0xe2500800SYSCALL,NX,MMX+,FFXSR,LM,3DNow!+,3DNow! AMD Features2=0x3LAHF,CMP real memory = 3758030848 (3583 MB) avail memory = 3677495296 (3507 MB) ACPI APIC Table: A M I OEMAPIC FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs FreeBSD/SMP: 4 package(s) x 2 core(s) ... 2. uname -v FreeBSD 9.0-CURRENT #3 3. sysctl kern.osreldate kern.osreldate: 900014 4. //depot/projects/soc2010/ringmap/ No help, but just curious - do use amd64 variant? If yes, can you reproduce the problem with i386? No, my kernel is i386, but I will try test it with amd64. Thanks Alex ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: coherence-problem on the mapped memory buffer
on 29/07/2010 19:13 Andriy Gapon said the following: on 29/07/2010 17:13 Alexander Fiveg said the following: P.S. Details about hardware and used software: 1. /var/run/dmesg.boot : ... CPU: Dual Core AMD Opteron(tm) Processor 865 (1800.01-MHz 686-class CPU) Origin = AuthenticAMD Id = 0x20f10 Family = f Model = 21 Stepping = 0 Features=0x178bfbffFPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT Features2=0x1SSE3 AMD Features=0xe2500800SYSCALL,NX,MMX+,FFXSR,LM,3DNow!+,3DNow! AMD Features2=0x3LAHF,CMP real memory = 3758030848 (3583 MB) avail memory = 3677495296 (3507 MB) ACPI APIC Table: A M I OEMAPIC FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs FreeBSD/SMP: 4 package(s) x 2 core(s) ... 2. uname -v FreeBSD 9.0-CURRENT #3 3. sysctl kern.osreldate kern.osreldate: 900014 4. //depot/projects/soc2010/ringmap/ In fact I have a suspicion that the problem might have to do with multiple mappings of the shared pages, but far from sure... Take a look at Intel® 64 and IA-32 Architectures Software Developer’s Manual Volume 3A - System Programming Guide, Part 1; Chapter 11.12.4 Programming the PAT; starting at the following words: «The PAT allows any memory type to be specified in the page tables, and therefore it is possible to have a single physical page mapped to two or more different linear addresses, each with different memory types. Intel does not support this practice...» -- Andriy Gapon ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: coherence-problem on the mapped memory buffer
on 29/07/2010 19:45 Alexander Fiveg said the following: On Thursday 29 July 2010 18:13:23 Andriy Gapon wrote: on 29/07/2010 17:13 Alexander Fiveg said the following: P.S. Details about hardware and used software: 1. /var/run/dmesg.boot : ... CPU: Dual Core AMD Opteron(tm) Processor 865 (1800.01-MHz 686-class CPU) Origin = AuthenticAMD Id = 0x20f10 Family = f Model = 21 Stepping = 0 Features=0x178bfbffFPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE, MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT Features2=0x1SSE3 AMD Features=0xe2500800SYSCALL,NX,MMX+,FFXSR,LM,3DNow!+,3DNow! AMD Features2=0x3LAHF,CMP real memory = 3758030848 (3583 MB) avail memory = 3677495296 (3507 MB) ACPI APIC Table: A M I OEMAPIC FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs FreeBSD/SMP: 4 package(s) x 2 core(s) ... 2. uname -v FreeBSD 9.0-CURRENT #3 3. sysctl kern.osreldate kern.osreldate: 900014 4. //depot/projects/soc2010/ringmap/ No help, but just curious - do use amd64 variant? If yes, can you reproduce the problem with i386? No, my kernel is i386, but I will try test it with amd64. Oh, nevermind actually. -- Andriy Gapon ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Re: coherence-problem on the mapped memory buffer
Jul 29, 2010 12:58:07 PM, a...@icyb.net.ua wrote: on 29/07/2010 19:13 Andriy Gapon said the following: on 29/07/2010 17:13 Alexander Fiveg said the following: In fact I have a suspicion that the problem might have to do with multiple mappings of the shared pages, but far from sure... Take a look at Intel® 64 and IA-32 Architectures Software Developer’s Manual Volume 3A - System Programming Guide, Part 1; Chapter 11.12.4 Programming the PAT; starting at the following words: «The PAT allows any memory type to be specified in the page tables, and therefore it is possible to have a single physical page mapped to two or more different linear addresses, each with different memory types. Intel does not support this practice...» My guess would be that the memory type is not marked as DMA-capable. AFAIK the Intel CPUs do the hardware snooping on the physical addresses, so they have no coherency issues benween themselves. However if a DMA writer changes the memory, this I think does not get normally propagated to the front-side bus, and the CPUs would not see it. You may need to either explicitly flush the CPU cache before accessing these pages or mark them as non-cacheable. -SB ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: coherence-problem on the mapped memory buffer
on 29/07/2010 23:02 Sergey Babkin said the following: Jul 29, 2010 12:58:07 PM, a...@icyb.net.ua wrote: on 29/07/2010 19:13 Andriy Gapon said the following: on 29/07/2010 17:13 Alexander Fiveg said the following: In fact I have a suspicion that the problem might have to do with multiple mappings of the shared pages, but far from sure... Take a look at Intel® 64 and IA-32 Architectures Software Developer’s Manual Volume 3A - System Programming Guide, Part 1; Chapter 11.12.4 Programming the PAT; starting at the following words: «The PAT allows any memory type to be specified in the page tables, and therefore it is possible to have a single physical page mapped to two or more different linear addresses, each with different memory types. Intel does not support this practice...» My guess would be that the memory type is not marked as DMA-capable. AFAIK the Intel CPUs do the hardware snooping on the physical addresses, so they have no coherency issues benween themselves. However if a DMA writer changes the memory, this I think does not get normally propagated to the front-side bus, and the CPUs would not see it. You may need to either explicitly flush the CPU cache before accessing these pages or mark them as non-cacheable. My guess was approximately the same - if one mapping is done in kernel for DMA purposes, then the memory type is, most likely, set to uncached. But the userland mapping of the same pages most likely marks the same pages (via different virtual addresses) as cached. Depending on the hardware and on what mappings were used on a particular CPU (core) to access that memory, there could be differences in interaction with DMA. -- Andriy Gapon ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: coherence-problem on the mapped memory buffer
On Thursday 29 July 2010 22:16:24 Andriy Gapon wrote: on 29/07/2010 23:02 Sergey Babkin said the following: Jul 29, 2010 12:58:07 PM, a...@icyb.net.ua wrote: on 29/07/2010 19:13 Andriy Gapon said the following: on 29/07/2010 17:13 Alexander Fiveg said the following: In fact I have a suspicion that the problem might have to do with multiple mappings of the shared pages, but far from sure... Take a look at Intel® 64 and IA-32 Architectures Software Developer’s Manual Volume 3A - System Programming Guide, Part 1; Chapter 11.12.4 Programming the PAT; starting at the following words: «The PAT allows any memory type to be specified in the page tables, and therefore it is possible to have a single physical page mapped to two or more different linear addresses, each with different memory types. Intel does not support this practice...» My guess would be that the memory type is not marked as DMA-capable. AFAIK the Intel CPUs do the hardware snooping on the physical addresses, so they have no coherency issues benween themselves. However if a DMA writer changes the memory, this I think does not get normally propagated to the front-side bus, and the CPUs would not see it. You may need to either explicitly flush the CPU cache before accessing these pages or mark them as non-cacheable. My guess was approximately the same - if one mapping is done in kernel for DMA purposes, then the memory type is, most likely, set to uncached. But the userland mapping of the same pages most likely marks the same pages (via different virtual addresses) as cached. Depending on the hardware and on what mappings were used on a particular CPU (core) to access that memory, there could be differences in interaction with DMA. Thanks a lot for your answers. But i am afraid i do not have enough experience to solve these tasks. Could you please provide me with helpful information how to: - get access to the pages associated with a certain memory-buffer ? I mean, I want to get the structures, that describe the page properties I should change (for instance, in order to make the page non-cacheable). if you are aware of any good papers or examples in the system code, where these topics are covered, I would appreciate it if you gave me the references. Alex ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: coherence-problem on the mapped memory buffer
on 30/07/2010 00:41 Alexander Fiveg said the following: Thanks a lot for your answers. But i am afraid i do not have enough experience to solve these tasks. Could you please provide me with helpful information how to: - get access to the pages associated with a certain memory-buffer ? I mean, I want to get the structures, that describe the page properties I should change (for instance, in order to make the page non-cacheable). if you are aware of any good papers or examples in the system code, where these topics are covered, I would appreciate it if you gave me the references. I don't have a recipe, but some pointers to get you started: 1. investigate BUS_DMA_NOCACHE, see bus_dma(9) 2. check sys/dev/sound/pci/hda/hdac.c for HDAC_F_DMA_NOCACHE and comment about PCIe snoop - this might be relevenat 3. see pmap_change_attr for way to change caching type for a memory mapping 4. hope that more knowledgeable people (experts) provide their advice, keep nudging them via mailing list(s) :-) -- Andriy Gapon ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
