----- Original Message ----- From: "Alexander Duyck" <alexander.du...@gmail.com> To: "psturm" <pst...@computervoice.com> Cc: "Emil S Tantilov" <emil.s.tanti...@intel.com>, "e1000-devel" <e1000-devel@lists.sourceforge.net> Sent: Sunday, May 8, 2016 10:16:53 PM Subject: Re: [E1000-devel] dmesg warnings for ixgbe on Xeon D-1540
On Sun, May 8, 2016 at 8:26 PM, Paul Sturm <pst...@computervoice.com> wrote: > ----- Original Message ----- > From: "Alexander Duyck" <alexander.du...@gmail.com> > To: "psturm" <pst...@computervoice.com> > Cc: "Emil S Tantilov" <emil.s.tanti...@intel.com>, "e1000-devel" > <e1000-devel@lists.sourceforge.net> > Sent: Tuesday, April 12, 2016 10:14:47 PM > Subject: Re: [E1000-devel] dmesg warnings for ixgbe on Xeon D-1540 > > On Tue, Apr 12, 2016 at 7:15 PM, Paul Sturm <pst...@computervoice.com> wrote: >> From: "Tantilov, Emil S" <emil.s.tanti...@intel.com> >> To: "psturm" <pst...@computervoice.com> >> Cc: "e1000-devel" <e1000-devel@lists.sourceforge.net> >> Sent: Wednesday, April 6, 2016 12:52:35 PM >> Subject: RE: dmesg warnings for ixgbe on Xeon D-1540 >> >>>>-----Original Message----- >>>>From: Paul Sturm [mailto:pst...@computervoice.com] >>>>Sent: Tuesday, April 05, 2016 10:46 AM >>>>To: Tantilov, Emil S <emil.s.tanti...@intel.com> >>>>Cc: e1000-devel <e1000-devel@lists.sourceforge.net> >>>>Subject: Re: dmesg warnings for ixgbe on Xeon D-1540 >>>> >>>>Perhaps "problem" was a poor choice of words. I have not yet begun testing >>>>the interface. I first wanted to see the driver load cleanly. So I am >>>>asking about both the ACPI warning and also the pmd_set_huge message. Does >>>>that clarify my request? >> >>>If there are any issues with the driver you would generally see a message >>>prefixed with "ixgbe:". For the ACPI warning - you may have better luck >>>getting an answer on a kernel mailing list. >>> >>>Thanks, >>>Emil >> >> Emil, >> >> I am happy to try to get a resolution on the ACPI issue from another list, >> but can you please assist with the hugepage/MTRR issue? I have built the >> driver into the kernel and also loaded it as a module. If I do not load the >> driver at all, I do not get the warning. If I build it in the kernel or load >> it as a module, I get the warning. I even set a kernel param for >> hugepagesz=1GB to make sure a huge page would be available to the driver. >> Here is the relevant snippet from the dmesg. I am attaching the full dmesg >> for your reference. >> >> Thanks, >> >> Paul >> >> [ 5453.184701] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - >> version 4 >> .2.1-k >> [ 5453.184704] ixgbe: Copyright (c) 1999-2015 Intel Corporation. >> [ 5453.184767] ACPI Warning: \_SB_.PCI0.BR2C._PRT: Return Package has no >> elements (empty) (20150930/nsprepkg-126) >> [ 5453.184891] pmd_set_huge: Cannot satisfy [mem >> 0x383fffa00000-0x383fffc00000] with a huge-page mapping due to MTRR override. >> [ 5454.248014] ACPI Warning: \_SB_.PCI0.BR2C._PRT: Return Package has no >> elements (empty) (20150930/nsprepkg-126) >> [ 5454.248077] ixgbe 0000:03:00.0: Multiqueue Enabled: Rx Queue count = 16, >> Tx Queue count = 16 >> [ 5454.318583] ixgbe 0000:03:00.0: MAC: 5, PHY: 6, PBA No: 020000-000 >> [ 5454.318585] ixgbe 0000:03:00.0: 0c:c4:7a:74:1a:21 >> [ 5454.345097] ixgbe 0000:03:00.0 eno1: renamed from eth2 >> [ 5454.357509] ixgbe 0000:03:00.0: Intel(R) 10 Gigabit Network Connection >> [ 5454.357680] ACPI Warning: \_SB_.PCI0.BR2C._PRT: Return Package has no >> elements (empty) (20150930/nsprepkg-126) >> [ 5455.935104] ACPI Warning: \_SB_.PCI0.BR2C._PRT: Return Package has no >> elements (empty) (20150930/nsprepkg-126) >> [ 5455.935167] ixgbe 0000:03:00.1: Multiqueue Enabled: Rx Queue count = 16, >> Tx Queue count = 16 >> [ 5456.005691] ixgbe 0000:03:00.1: MAC: 5, PHY: 6, PBA No: 020000-000 >> >> Earlier you see: >> [ 0.325307] HugeTLB registered 1 GB page size, pre-allocated 0 pages >> [ 0.325315] HugeTLB registered 2 MB page size, pre-allocated 0 pages >> >> I also tried pre-allocating both 1GB and 2MB HugeTLBs, but the driver always >> wants to go to a memory area that the MTRR seems to have overriden. So there >> are HugeTLBs available, just not where the driver wants to look apparently? > > Paul, > > You aren't going to find a bug in the driver causing any issues. You > would be much better off taking this onto the linux-mm list and asking > there. > > I'm suspecting this is expected behavior due to the fact that you are > limited to 4K pages for regions that are defined as being UC via the > MTRR, or at least that is my understanding of how this ends up working > out. You might reference the patch set that enabled huge pages for IO > for more information, https://lwn.net/Articles/635357/ > > - Alex > > Alex, > > I did some further investigation. It appears that the warning comes about as > a result of this commit: > > https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/?h=linux-4.2.y&id=b73522e0c1be58d3c69b124985b8ccf94e3677f7 > > From the log entry: > > "pmd_set_huge() logs a warning message to a non-optimal request > so that driver writers will be aware of such a case. Drivers > should make a mapping request aligned to a single MTRR entry > when the range is covered by MTRRs." > > So my question is whether the ixgbe driver is in fact making a mapping > request aligned as described in the log entry for this commit? > > Using iperf and testing kernel 4.1.21 versus 4.2 using ixgbe version 4.3.15: > > command line: iperf3 -c10.10.30.2 -p5201 -w40k -n%G -b10G -u > > Execution time regresses from 18.3 to 18.46. > Bandwidth regresses from 2.35 to 2.33 > > using perf stat for the above test: > > page faults increase from 45 to 60 (pmd_set_huge is one of the new warnings > that appear in kernel 4.2) > > instructions per clock decreased from .6 to .56 > > Paul Most of these numbers don't mean much. A < 1% variance between two different kernel versions could be anything. The warning message indicates that it couldn't use a huge page mapping for the MMIO region BAR 0. If you check lspci -vvv for the device you should be able to tell if the region in the message happens to correspond to BAR 0, or some other region for the device. - Alex Alex, I wanted to let you know the issue is resolved. I referred to the article you mentioned above on lwn.net. An uncacheable MTRR was overlapping the 2MB region that ixgbe needed to map to for the 10g ethernet interface. After studying what MTRR was used for, I realized it was kind of a legacy feature from the days of AGP and PCI graphics. I wasn't able to remove it from my kernel builds until I realized that MTRR suppport depended on CONFIG_EXPERT. Once I was able to remove MTRR from the kernel, the overlap disappeared and pmd_set_huge was able to grab the memory it wanted. No more warning. Voila! Thanks for your patience. I appreciate it. Paul ------------------------------------------------------------------------------ Mobile security can be enabling, not merely restricting. Employees who bring their own devices (BYOD) to work are irked by the imposition of MDM restrictions. Mobile Device Manager Plus allows you to control only the apps on BYO-devices by containerizing them, leaving personal data untouched! https://ad.doubleclick.net/ddm/clk/304595813;131938128;j _______________________________________________ E1000-devel mailing list E1000-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
