RE: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-11 Thread Pravin Bathija
Stefan Roese wrote: Correct. IIRC, some PATA driver as Pravin already mentioned. Cheers, Stefan Thanks Stefan. The whole intention of the patch/hack (or whatever one might call it :) ) was to avoid rogue drivers from setting pci_cache_line_size to non-zero value even though the underlying

Re: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-11 Thread Mikhail Zolotaryov
Benjamin Herrenschmidt wrote: On Wed, 2009-09-09 at 17:40 +0300, Mikhail Zolotaryov wrote: Hi Tom, In my case __dma_sync() calls flush_dcache_range() (it's due to alignment) from a tasklet - no OOPS. It uses dcbf instruction instead of dcbi - that's the difference as dcbf is not

Re: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-11 Thread Benjamin Herrenschmidt
On Fri, 2009-09-11 at 10:17 +0300, Mikhail Zolotaryov wrote: Benjamin Herrenschmidt wrote: On Wed, 2009-09-09 at 17:40 +0300, Mikhail Zolotaryov wrote: Hi Tom, In my case __dma_sync() calls flush_dcache_range() (it's due to alignment) from a tasklet - no OOPS. It uses dcbf

Re: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-11 Thread Tom Burns
Hi Ben, Benjamin Herrenschmidt wrote: On Wed, 2009-09-09 at 09:43 -0400, Tom Burns wrote: Hi, With the default config for the Sequoia board on 2.6.24, calling pci_dma_sync_sg_for_cpu() results in executing invalidate_dcache_range() in arch/ppc/kernel/misc.S from __dma_sync(). This

RE: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-11 Thread Prodyut Hazarika
I tried, using our JTAG debugger (BDI3000), to pause operation after calling dma_alloc_coherent to examine the TLB entry for the memory returned by the call (which was just past CONFIG_CONSISTENT_START=0xff10). The TLB list loaded at the time that I paused operation did not show a

RE: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-11 Thread Pravin Bathija
Benjamin Herrenschmidt wrote: Do you know many drivers that do config space accesses without using the config space accessors ? Such drivers should be banned to oblivion. Cheers, Ben. I'm not aware of such drivers in the 2.6.30+ kernel. Thanks, Pravin On Thu, 2009-09-10 at 22:35 -0700,

Re: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-10 Thread Tom Burns
Hi, Thank you everyone for your help. I've been looking into the other dma/pci API calls (dma_alloc_coherent, pci_alloc_consistent). I don't see how either of these return memory mapped to a TLB with the I bit set to 1 in kernel 2.6.24. In our kernel code, the only use of the PPC44x_TLB_I

RE: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-10 Thread Pravin Bathija
Tom Burns wrote Hi, Thank you everyone for your help. I've been looking into the other dma/pci API calls (dma_alloc_coherent, pci_alloc_consistent). I don't see how either of these return memory mapped to a TLB with the I bit set to 1 in kernel 2.6.24. In our kernel code, the only

Re: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-10 Thread Benjamin Herrenschmidt
On Wed, 2009-09-09 at 10:10 -0400, Tom Burns wrote: Hi Mikhail, Sorry, this DMA code is in a tasklet. Are you suggesting the processor is in supervisor mode at that time? Calling pci_dma_sync_sg_for_cpu() from the tasklet context is what generates the OOPS. The entire oops is as

Re: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-10 Thread Benjamin Herrenschmidt
On Wed, 2009-09-09 at 17:40 +0300, Mikhail Zolotaryov wrote: Hi Tom, In my case __dma_sync() calls flush_dcache_range() (it's due to alignment) from a tasklet - no OOPS. It uses dcbf instruction instead of dcbi - that's the difference as dcbf is not privileged. What it calls depends on

Re: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-10 Thread Benjamin Herrenschmidt
On Thu, 2009-09-10 at 15:53 -0400, Tom Burns wrote: Hi, Thank you everyone for your help. I've been looking into the other dma/pci API calls (dma_alloc_coherent, pci_alloc_consistent). I don't see how either of these return memory mapped to a TLB with the I bit set to 1 in kernel

RE: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-10 Thread Benjamin Herrenschmidt
On Thu, 2009-09-10 at 13:30 -0700, Pravin Bathija wrote: There is also a patch that was submitted for 440EPX a couple of years back. The 440EPX SOC causes hangs with Memory Read Multiple (MRM) commands. Whether MRM is used or not depends on the value of PCI_CACHE_LINE_SIZE register. I see

Re: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-10 Thread Stefan Roese
On Friday 11 September 2009 04:44:34 Benjamin Herrenschmidt wrote: On Thu, 2009-09-10 at 13:30 -0700, Pravin Bathija wrote: There is also a patch that was submitted for 440EPX a couple of years back. The 440EPX SOC causes hangs with Memory Read Multiple (MRM) commands. Whether MRM is used

Re: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-10 Thread Benjamin Herrenschmidt
On Fri, 2009-09-11 at 07:12 +0200, Stefan Roese wrote: It's already there. See commit: 5ce4b59653b2c2053cd9a011918ac1e4747f24cc powerpc/4xx: Workaround for PPC440EPx/GRx PCI_28 Errata Ok, that's another way to do it. Will catch nasty drivers who try to write directly rather than clear

Re: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-10 Thread Stefan Roese
On Friday 11 September 2009 07:17:50 Benjamin Herrenschmidt wrote: On Fri, 2009-09-11 at 07:12 +0200, Stefan Roese wrote: It's already there. See commit: 5ce4b59653b2c2053cd9a011918ac1e4747f24cc powerpc/4xx: Workaround for PPC440EPx/GRx PCI_28 Errata Ok, that's another way to do it.

RE: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-10 Thread Benjamin Herrenschmidt
On Thu, 2009-09-10 at 22:35 -0700, Pravin Bathija wrote: Thanks Stefan. The whole intention of the patch/hack (or whatever one might call it :) ) was to avoid rogue drivers from setting pci_cache_line_size to non-zero value even though the underlying hardware doesn't support MRM calls.

Re: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-09 Thread Mikhail Zolotaryov
Hi, Why manage cache lines manually, if appropriate code is a part of __dma_sync / dma_sync_single_for_device of DMA API ? (implies CONFIG_NOT_COHERENT_CACHE enabled, as default for Sequoia Board) Prodyut Hazarika wrote: Hi Adam, Yes, I am using the 440EPx (same as the sequoia board).

Re: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-09 Thread Mikhail Zolotaryov
Hi Tom, possible solution could be to use tasklet to perform DMA-related job (as in most cases DMA transfer is interrupt driven - makes sense). Tom Burns wrote: Hi, With the default config for the Sequoia board on 2.6.24, calling pci_dma_sync_sg_for_cpu() results in executing

Re: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-09 Thread Tom Burns
Hi, With the default config for the Sequoia board on 2.6.24, calling pci_dma_sync_sg_for_cpu() results in executing invalidate_dcache_range() in arch/ppc/kernel/misc.S from __dma_sync(). This OOPses on PPC440 since it tries to call directly the assembly instruction dcbi, which can only be

Re: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-09 Thread Tom Burns
Hi Mikhail, Sorry, this DMA code is in a tasklet. Are you suggesting the processor is in supervisor mode at that time? Calling pci_dma_sync_sg_for_cpu() from the tasklet context is what generates the OOPS. The entire oops is as follows, if it's relevant: Oops: kernel access of bad area,

RE: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-08 Thread Adam Zilkie
Hi Prodyut, Yes, I am using the 440EPx (same as the sequoia board). Our ideDriver is DMA'ing blocks of 192-byte data over the PCI bus (using the Sil0680A PCI-IDE bridge). Most of the DMA's (depending on timing) end up being partially corrupted when we try to parse the data in the virtual page.

RE: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-08 Thread Prodyut Hazarika
Hi Adam, Yes, I am using the 440EPx (same as the sequoia board). Our ideDriver is DMA'ing blocks of 192-byte data over the PCI bus (using the Sil0680A PCI-IDE bridge). Most of the DMA's (depending on timing) end up being partially corrupted when we try to parse the data in the virtual page.

RE: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-08 Thread Prodyut Hazarika
We have found that using flush_dcache_range() after each DMA solves the problem. Ideally, we'd like to be able to allocate the virtual page in cache inhibited memory to avoid the performance loss from all the flush calls. To do this, we'd have to change our TLB sizes and reserve a TLB in

RE: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-08 Thread Benjamin Herrenschmidt
On Tue, 2009-09-08 at 14:01 -0400, Adam Zilkie wrote: Hi Prodyut, Yes, I am using the 440EPx (same as the sequoia board). Our ideDriver is DMA'ing blocks of 192-byte data over the PCI bus (using the Sil0680A PCI-IDE bridge). Most of the DMA's (depending on timing) end up being partially

RE: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-08 Thread Benjamin Herrenschmidt
On Tue, 2009-09-08 at 15:30 -0400, Adam Zilkie wrote: All, We have found that using flush_dcache_range() after each DMA solves the problem. Ideally, we'd like to be able to allocate the virtual page in cache inhibited memory to avoid the performance loss from all the flush calls. To do

RE: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-08 Thread Adam Zilkie
We are using pci_alloc_consistent() Adam On Tue, 2009-09-08 at 12:56 -0700, Prodyut Hazarika wrote: We have found that using flush_dcache_range() after each DMA solves the problem. Ideally, we'd like to be able to allocate the virtual page in cache inhibited memory to avoid the

RE: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-08 Thread Adam Zilkie
All, We have found that using flush_dcache_range() after each DMA solves the problem. Ideally, we'd like to be able to allocate the virtual page in cache inhibited memory to avoid the performance loss from all the flush calls. To do this, we'd have to change our TLB sizes and reserve a TLB in

RE: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-08 Thread Benjamin Herrenschmidt
On Tue, 2009-09-08 at 16:00 -0400, Adam Zilkie wrote: We are using pci_alloc_consistent() Then your flush should have no effect since pci_alloc_consistent will return I=1 mapped memory, unless you don't have CONFIG_NOT_COHERENT_CACHE for some reason. Cheers, Ben. Adam On Tue, 2009-09-08 at

RE: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-06 Thread Benjamin Herrenschmidt
; linuxppc-dev@lists.ozlabs.org Subject: Re: AW: PowerPC PCI DMA issues (prefetch/coherency?) Hi Adam, If you have a look in include/asm-ppc/pgtable.h for the following section: #ifdef CONFIG_44x #define _PAGE_BASE(_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_GUARDED) #else #define

Re: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-03 Thread Chris Pringle
Hi Adam, If you have a look in include/asm-ppc/pgtable.h for the following section: #ifdef CONFIG_44x #define _PAGE_BASE(_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_GUARDED) #else #define _PAGE_BASE(_PAGE_PRESENT | _PAGE_ACCESSED) #endif Try adding _PAGE_COHERENT to the appropriate line

Re: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-03 Thread Benjamin Herrenschmidt
On Thu, 2009-09-03 at 09:05 +0100, Chris Pringle wrote: Hi Adam, If you have a look in include/asm-ppc/pgtable.h for the following section: #ifdef CONFIG_44x #define _PAGE_BASE(_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_GUARDED) #else #define _PAGE_BASE(_PAGE_PRESENT | _PAGE_ACCESSED)

Re: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-03 Thread Chris Pringle
; linuxppc-dev@lists.ozlabs.org Subject: Re: AW: PowerPC PCI DMA issues (prefetch/coherency?) Hi Adam, If you have a look in include/asm-ppc/pgtable.h for the following section: #ifdef CONFIG_44x #define _PAGE_BASE(_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_GUARDED) #else #define _PAGE_BASE

RE: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-03 Thread Wrobel Heinz-R39252
DMA issues (prefetch/coherency?) Hi Adam, If you have a look in include/asm-ppc/pgtable.h for the following section: #ifdef CONFIG_44x #define _PAGE_BASE(_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_GUARDED) #else #define _PAGE_BASE(_PAGE_PRESENT | _PAGE_ACCESSED) #endif Try adding

Re: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-03 Thread Adam Zilkie
Chris, I noticed the following comment in pgtable.h: * - CACHE COHERENT bit (M) has no effect on PPC440 core, because it * doesn't support SMP. So we can use this as software bit, like * DIRTY. And _PAGE_COHERENT is not defined for the 44x (giving a compile error when I add it the

Re: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-03 Thread Adam Zilkie
Ben, Thanks for your info. Are you sure there is L2 cache on the 440? I am seeing this problem with our custom IDE driver which is based on pretty old code. Our driver uses pci_alloc_consistent() to allocate the physical DMA memory and alloc_pages() to allocate a virtual page. It then uses

Re: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-03 Thread Josh Boyer
On Thu, Sep 03, 2009 at 12:04:50PM -0400, Adam Zilkie wrote: Ben, Thanks for your info. Are you sure there is L2 cache on the 440? It depends on which 440 SoC you have. It also depends on that being configured in the kernel even if it does exist. I am seeing this problem with our custom IDE

RE: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-03 Thread Prodyut Hazarika
Hi Adam, Are you sure there is L2 cache on the 440? It depends on the SoC you are using. SoC like 460EX (Canyonlands board) have L2Cache. It seems you are using a Sequoia board, which has a 440EPx SoC. 440EPx has a 440 cpu core, but no L2Cache. Could you please tell me which SoC you are using?

Re: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-09-02 Thread Adam Zilkie
Hi Chris, I am having a problem similar to what you described in this discussion. We are using the ppc arch with 2.6.24 with CONFIG_SEQUOIA with compiles arch/ppc/kernel/head_44x.c (quite different from /arch/powerpc/kernel/head_32.S). I would like to apply your backporting patch to this

Re: AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-06-29 Thread Chris Pringle
Hi Sergej, I've attached the patch used to fix this issue. Both the patch to pgtable32.h and head_32.S are required in order to make it work. The change to pgtable32.h ensures that all pages are marked cache coherent (results in setting the M bit). The change to head_32.S ensures that the M

AW: PowerPC PCI DMA issues (prefetch/coherency?)

2009-06-22 Thread Sergej.Stepanov
The other part of the fix is in asm-powerpc/pgtable32.h. _PAGE_BASE needs _PAGE_COHERENT in order to work correctly, and in fact there is now a comment in there to that affect in 2.6.29. Backporting that change has made it work on 2.6.26. Both this patch, and the fix to head_32.S are needed for it

Re: PowerPC PCI DMA issues (prefetch/coherency?)

2009-06-18 Thread Chris Pringle
Chris Pringle wrote: Chris Pringle wrote: You could enable CONFIG_NOT_COHERENT_CACHE. I've just tried this (I had to edit Kconfig in power/platforms to make the build system accept it), and interestingly it's making no difference. I'm using streaming mappings, and are using the pci_map_sg

Re: PowerPC PCI DMA issues (prefetch/coherency?)

2009-06-17 Thread Chris Pringle
You could enable CONFIG_NOT_COHERENT_CACHE. I've just tried this (I had to edit Kconfig in power/platforms to make the build system accept it), and interestingly it's making no difference. I'm using streaming mappings, and are using the pci_map_sg functions to ensure the memory is

Re: PowerPC PCI DMA issues (prefetch/coherency?)

2009-06-17 Thread Chris Pringle
Chris Pringle wrote: You could enable CONFIG_NOT_COHERENT_CACHE. I've just tried this (I had to edit Kconfig in power/platforms to make the build system accept it), and interestingly it's making no difference. I'm using streaming mappings, and are using the pci_map_sg functions to ensure the

PowerPC PCI DMA issues (prefetch/coherency?)

2009-06-16 Thread Chris Pringle
Hello All, We're developing on a Freescale MPC8272 and are having some nasty problems with PCI bus mastering and data corruption. We have some custom hardware that is bus mastering, reading data from the CPUs memory for it's own use. Most of the time, the data is correct, however

Re: PowerPC PCI DMA issues (prefetch/coherency?)

2009-06-16 Thread Roderick Colenbrander
Hi Chris, The last couple of months I had the 'pleasure' to work with a Xilinx ML510 ATX board which contains a Virtex-5 FXT. In my case I had to set up the plbv46 pci soft-core to function properly with PCI add-on boards and onboard pci devices. I have had a lot of issues including ones with DMA

Re: PowerPC PCI DMA issues (prefetch/coherency?)

2009-06-16 Thread Scott Wood
Chris Pringle wrote: The kernel version is 2.6.26. Firmware is custom on a custom board. Cache coherency on PCI DMA requires that the memory be mapped with the M attribute on this chip, but that should be happening based on detection of the core. I'm not sure where to look to verify this?

Re: PowerPC PCI DMA issues (prefetch/coherency?)

2009-06-16 Thread Chris Pringle
Scott Wood wrote: Check asm/cputable.h for CPU_FTR_NEED_COHERENT. Make sure that CONFIG_8260 is one of the #ifdefs that turns that on. It looks like that was in place by 2.6.26 in arch/powerpc. I'm not sure what to look for in arch/ppc. I've just checked that and it's definitely switched

Re: PowerPC PCI DMA issues (prefetch/coherency?)

2009-06-16 Thread Scott Wood
Chris Pringle wrote: Ah right - that would explain what we're seeing then... Doh. Thought I might have been onto something then. Is there any way to force a cache flush? That'd at least prove it was a caching issue if it resolved the problem. You could enable CONFIG_NOT_COHERENT_CACHE.

Re: PowerPC PCI DMA issues (prefetch/coherency?)

2009-06-16 Thread Scott Wood
Arnd Bergmann wrote: On Tuesday 16 June 2009, Scott Wood wrote: Chris Pringle wrote: Ah right - that would explain what we're seeing then... Doh. Thought I might have been onto something then. Is there any way to force a cache flush? That'd at least prove it was a caching issue if it resolved

Re: PowerPC PCI DMA issues (prefetch/coherency?)

2009-06-16 Thread Arnd Bergmann
On Tuesday 16 June 2009, Scott Wood wrote: If the device is the only one, you can also use dma_alloc_noncoherent() and flush explicitly with dma_cache_sync(). I don't see how that would help -- aren't those also controlled by CONFIG_NOT_COHERENT_CACHE? Ah, yes you are right. PowerPC

Re: PowerPC PCI DMA issues (prefetch/coherency?)

2009-06-16 Thread Benjamin Herrenschmidt
On Tue, 2009-06-16 at 20:02 +0200, Arnd Bergmann wrote: On Tuesday 16 June 2009, Scott Wood wrote: If the device is the only one, you can also use dma_alloc_noncoherent() and flush explicitly with dma_cache_sync(). I don't see how that would help -- aren't those also controlled by

Re: PowerPC PCI DMA issues (prefetch/coherency?)

2009-06-16 Thread FUJITA Tomonori
On Wed, 17 Jun 2009 10:18:45 +1000 Benjamin Herrenschmidt b...@kernel.crashing.org wrote: On Tue, 2009-06-16 at 20:02 +0200, Arnd Bergmann wrote: On Tuesday 16 June 2009, Scott Wood wrote: If the device is the only one, you can also use dma_alloc_noncoherent() and flush explicitly

Re: PowerPC PCI DMA issues (prefetch/coherency?)

2009-06-16 Thread Leon Woestenberg
Hello all, On Wed, Jun 17, 2009 at 2:37 AM, FUJITA Tomonorifujita.tomon...@lab.ntt.co.jp wrote: On Wed, 17 Jun 2009 10:18:45 +1000 Benjamin Herrenschmidt b...@kernel.crashing.org wrote: On Tue, 2009-06-16 at 20:02 +0200, Arnd Bergmann wrote: On Tuesday 16 June 2009, Scott Wood wrote: If

Re: PowerPC PCI DMA issues (prefetch/coherency?)

2009-06-16 Thread Benjamin Herrenschmidt
On Wed, 2009-06-17 at 09:37 +0900, FUJITA Tomonori wrote: dma_cache_sync is supposed to be used only with the buffers that dma_alloc_noncoherent() returns. On architecutures that maps dma_alloc_noncoherent to dma_alloc_coherent, dma_cache_sync() is supposed to be NOP. Or at least a sync()