Hi Ben, The second issue. you told me "some hosts" has problem, and some hosts worked well. what is the problem-hosts? The issue seems like the hosts did set the NO SNOOP attribute bit at TLP. The PEX_DEVICE_CONTROL is standard PCI configuration space register, it controls the behavior of the initiator's transaction. For 8315, it is outbound, not inbound transaction. Thanks, Dave
________________________________ From: Ben Menchaca [mailto:ben.mench...@gmail.com] Sent: Saturday, March 07, 2009 12:30 AM To: Liu Dave-R63238 Cc: linuxppc-dev@ozlabs.org Subject: Re: 83xx: Marking or Allocating Pages as Cache-Inhibited Thank you for your help! That bit resolved all of the RDMA/WDMA coherency issues on the CSB side...except: We expose a 1MB region of memory from CSB via a BAR (BAR1, if it matters) to the Host. This region is also not behaving correctly with respect to coherency on SOME hosts; again, disabling our cache makes it work correctly on all hosts. We have set PEX_DEVICE_CONTROL in PCI-E Config Space (0x54) to 0x2010 (sorry about the endianness below). We thought that CLEARING the no-snoop bit here would indicate that snooping was required for this region...is this a similar issue? - Ben On Fri, Mar 6, 2009 at 10:12 AM, Ben Menchaca <ben.mench...@gmail.com> wrote: Testing now...it looks like it (almost) works, though! Why does setting no-snoop cause snooping to work? More on the effect on setting that bit in a few minutes...need more testing. ACR is 0x00030300. - Ben On Fri, Mar 6, 2009 at 12:30 AM, Liu Dave-R63238 <dave...@freescale.com> wrote: Did you enable the descriptor bit 3 to have a try? ________________________________ From: Ben Menchaca [mailto:ben.mench...@gmail.com] Sent: Friday, March 06, 2009 2:10 PM To: Liu Dave-R63238 Cc: linuxppc-dev@ozlabs.org Subject: Re: 83xx: Marking or Allocating Pages as Cache-Inhibited I can look at ACR morning...although I can say with a fair amount of certainty that I have not changed it from the POR value. I will try enabling No Snoop for CSB in the descriptor (bit 3, yes?)...this seems a bit counterintuitive to me. What is the hope regarding these two? Some combination I am not seeing? On Thu, Mar 5, 2009 at 11:40 PM, Liu Dave-R63238 <dave...@freescale.com> wrote: what is the value of ACR register? ________________________________ From: Ben Menchaca [mailto:ben.mench...@gmail.com] Sent: Friday, March 06, 2009 1:38 PM To: Liu Dave-R63238 Cc: linuxppc-dev@ozlabs.org Subject: Re: 83xx: Marking or Allocating Pages as Cache-Inhibited 1. BAT2 in linux is set to WIMG=0010, and covers all 64M 2. PEX_DEVICE_CONTROL in PCI-E Config Space (0x54): 0x1020 3. PEX_xDMA_CTRL is set to 0x00000401 at the initiation of the DMA. 4. OWAR0 is set to 0xFFFFF005, so NSNP is 0. 5. The DMA descriptor (randomly chosen when I hit a trigger...just ignore the size...) contains 0002AFF3 at offset 0, so nosnoops are cleared. Core is 400MHz, and CSB is 133MHz. - Ben On Thu, Mar 5, 2009 at 11:27 PM, Liu Dave-R63238 <dave...@freescale.com> wrote: and what settings is DMA description bit 3? > -----Original Message----- > From: linuxppc-dev-bounces+daveliu=freescale....@ozlabs.org > [mailto:linuxppc-dev-bounces+daveliu <mailto:linuxppc-dev-bounces%2Bdaveliu> =freescale....@ozlabs.org] > On Behalf Of Liu Dave-R63238 > Sent: Friday, March 06, 2009 1:22 PM > To: Ben Menchaca; linuxppc-dev@ozlabs.org > Subject: RE: 83xx: Marking or Allocating Pages as Cache-Inhibited > > Did you enable the snoop bit at PEX_WDMA_CTRL[SNOOP] and > PEX_RDMA_CTRL[SNOOP]? > > What is the freq settings? CORE/CSB bus. > > Thanks, Dave > > ________________________________ > > From: linuxppc-dev-bounces+daveliu=freescale....@ozlabs.org > [mailto:linuxppc-dev-bounces+daveliu <mailto:linuxppc-dev-bounces%2Bdaveliu> =freescale....@ozlabs.org] > On Behalf Of Ben Menchaca > Sent: Friday, March 06, 2009 12:33 PM > To: linuxppc-dev@ozlabs.org > Subject: 83xx: Marking or Allocating Pages as Cache-Inhibited > > > I am working on a Freescale 8314e design, and the > embedded device is configured as a PCI-e endpoint running a > 2.6.27-5 kernel. For context, we have written a kernel > module which, among other things, uses the RDMA/WDMA engine > in the PCI-e IP block. On the host side, these DMAs are > coherent. However, on the embedded side, things are quite a > bit less rosy; we must manually flush/invalidate cache lines > for WDMA/RDMAs to occur successfully. After speaking with > (several) FAEs at Freescale, we believe there is a > configuration issue that is the cause, but we have yet to > have anyone successfully point to it. > > Disabling the data cache altogether resolves the issue > entirely, but of course, also completely tanks performance. > As a temporary workaround, I would like to simply mark the > pages (obtained currently via dma_alloc_coherent) involved as > cache-inhibited. I have attempted to do this via some > snippets remaining in fec.c (va_to_pte, uncache_pte to set > _PAGE_NO_CACHE, flush_tlb_page, then unmap_pte), but this is > almost certainly braindead; va_to_pte is not a part of the > 83xx source, as far as I can tell; 8xx only. > > A quick pointer in the correct direction for marking > pages as cache-inhibited on a 2.6.27-5 kernel would be > appreciated, or if my approach to a workaround is flawed, a > pointer to the correct way would be great. > > Ben Menchaca > > > _______________________________________________ > Linuxppc-dev mailing list > Linuxppc-dev@ozlabs.org > https://ozlabs.org/mailman/listinfo/linuxppc-dev > >
_______________________________________________ Linuxppc-dev mailing list Linuxppc-dev@ozlabs.org https://ozlabs.org/mailman/listinfo/linuxppc-dev