Please do check to see if the fence is fixing the problem. If it is not the
fence and it turns out it is some other delay then we can follow up.
Thanks,
- Greg
From: Brandon Falk [mailto:bf...@gamozolabs.com]
Sent: Wednesday, April 22, 2015 2:41 PM
To: Rose, Gregory V
Cc: e1000-devel@lists.sourceforge.net
Subject: Re: [E1000-devel] [non-linux] Cache Coherency Problems on X540?
I can see that there is a read memory barrier in the ixgbe_clean_rx_irq()
functions with the comment "This memory barrier is needed to keep us from
reading any other fields out of the rx_desc until we know the RXD_STAT_DD bit
is set" which is exactly my problem, but that should emit an lfence which if I
remember correctly did not happen to fix the problem for me.
I'll take a look when I get home. I can make a simple test to try to determine
if the fence is actually fixing the problem, or if it is just adding a delay
which makes the problem seem to disappear. Since I have a very lightweight and
lockless RX implementation, if it is a hardware bug it is unlikely to show up
in a 'proper' driver given they just happen to do enough logic between the DD
check and reading packet contents.
My code looks as such:
.poll:
-B
On Wed, Apr 22, 2015 at 5:04 PM, Rose, Gregory V
<gregory.v.r...@intel.com<mailto:gregory.v.r...@intel.com>> wrote:
> -----Original Message-----
> From: Brandon Falk [mailto:bf...@gamozolabs.com<mailto:bf...@gamozolabs.com>]
> Sent: Wednesday, April 22, 2015 1:10 PM
> To:
> e1000-devel@lists.sourceforge.net<mailto:e1000-devel@lists.sourceforge.net>
> Subject: [E1000-devel] [non-linux] Cache Coherency Problems on X540?
>
> I'm writing a custom driver for a custom OS and thus this does not relate
> to the Linux driver.
>
> I have implemented a driver which polls for the DD and EOP flags to get
> set (using legacy receive descriptors) and I have noticed a problem where
> these flags both can get set very slightly before the packet contents have
> actually been DMAed to the host (or at least appear to be).
>
> I'm under the understanding that the X540 device should respect caching
> and thus I should not have to be flushing caches for the packet contents.
> The only thing I could think of would be that I need to add some memory
> barriers as maybe the CPU is speculatively loading the packet contents
> before it is reading the flags, and thus the contents are older than the
> flags?
The Intel Linux driver for the X540 does implement memory barriers and I'm
pretty sure it's to prevent this sort of problem. You may want to examine that
code (understanding at the same time that GPL will prevent you from using it
directly).
- Greg
----------
Greg Rose
FreeBSD/NFV PAE
Network Division
Intel Corporation
Desk - 503-712-5048<tel:503-712-5048>
Any man who afflicts the human race with ideas must be prepared to see them
misunderstood.
- H. L. Mencken
>
> I just want to verify that this isn't a hardware problem with the X540 and
> that I will need to do a workaround on my end. Ideally the workaround
> actually fixes the problem, rather than a small delay/sleep that makes it
> seem to go away in most cases.
>
> Anyone have any ideas? Adding an clflush, mfence, sfence fixes the
> problem, but an lfence does not. I'm under the suspicion that it's not the
> fences that actually are doing anything, rather the delay of these rather
> long instructions happens to just be a long enough sleep. I just want to
> make sure that I'm fixing the problem correctly and not just hoping the
> sleep delays long enough for the DMA to occur, as that is quite kludgey.
>
> It is a 4 physical processor system, AMD Fam 15h.
>
> -B
------------------------------------------------------------------------------
BPM Camp - Free Virtual Workshop May 6th at 10am PDT/1PM EDT
Develop your own process in accordance with the BPMN 2 standard
Learn Process modeling best practices with Bonita BPM through live exercises
http://www.bonitasoft.com/be-part-of-it/events/bpm-camp-virtual- event?utm_
source=Sourceforge_BPM_Camp_5_6_15&utm_medium=email&utm_campaign=VA_SF
_______________________________________________
E1000-devel mailing list
E1000-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit
http://communities.intel.com/community/wired
