Hi Darrell, /Billy
> -----Original Message----- > From: Darrell Ball [mailto:db...@vmware.com] > Sent: Thursday, March 30, 2017 8:51 PM > To: O Mahony, Billy <billy.o.mah...@intel.com>; Brad Cowie > <b...@cowie.nz>; Shivaram Mysore <shivaram.mys...@gmail.com> > Cc: ovs-discuss@openvswitch.org > Subject: Re: [ovs-discuss] Error attaching device to DPDK > > > > On 3/29/17, 7:40 AM, "O Mahony, Billy" <billy.o.mah...@intel.com> wrote: > > Hi All, > > > > Just to add something to this conversation that has not been explicitly > mentioned below. > > > > Brad outlines how to set other_config:dpdk-socket-mem to reserve > hugepages from both NUMA nodes in order to be sure to avoid pci topology > issues. > > > > Strictly this doesn’t have to be done on both nodes – just on the correct > node :) – but sometimes it’s easier to just cover all the bases. > > > > In addition you will need to ensure that: > > * That linux has hugepages available on both (or the right) nodes. > > * There is a pmd thread created on each node by setting > other_config:pmd-cpu-mask correctly – or again at least on the right node. > > > > These config items are described in > > * ./Documentation/intro/install/dpdk.rst > > * ./Documentation/howto/dpdk.rst > > > > Currently if there is not a pmd thread on the NUMA node belonging to the > PCI device the device will not be polled (a warning message is issued to this > effect). > > > [Darrell] > From Linux POV, the “default” memory policy is to allocate the hugepage > pool across numa nodes, if possible; this assumes an allocation was done, > which we have no reports here otherwise. > iirc based on the logs in this case, which I cannot view anymore, there was no > pmd mask configuration specified. > The advertised behavior should be, in this case, that a pmd thread is > allocated on each numa node with at least one local dpdk interface bound. > iirc, the dpdk socket memory was left at default, which only pre-allocated > from the hugepage pool on node 0 So, does it make sense to also try to pre- > allocate hugepages on a numa node, where a PMD thread is allocated ? > Pls correct me in any respects. > [[BO'M]] What you're looking for here is something that will work (even if not optimally). And leave the optimization of exact configurations of pmds/hugepage-allocations and so on until later? So even if pmd-cpu-mask and dpdk-socket-mem other_config items are omitted OVS will 'do the right thing' ? ie. place a pmd on each numa node and take some hugepages on each NUMA node. It's hard to argue with that and I can put that on our internal backlog of tasks. How about these rules? * If pmd-cpu-mask is not set OVS creates one pmd thread for each NUMA node. (current case) * if dpdk-socket-mem is not set then for each NUMA node dpdk will reserve: ** one quarter of the available hugepage memory ** 512MB of 2MB hugepages If hugepage granularity requires more memory may be reserved. > > > > > There is currently a patch under consideration so that if a NUMA-local pmd > is not available for a device then device will be assigned to some other pmd > thread. While this is not as efficient due to the need for the data to travel > between NUMA nodes it is less frustrating to configure. > > > > https://urldefense.proofpoint.com/v2/url?u=https- > 3A__mail.openvswitch.org_pipermail_ovs-2Ddev_2017- > 2DFebruary_329175.html&d=DwIGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=BVh > FA09CGX7JQ5Ih- > uZnsw&m=IKHiqOFj5oogxEfx6OMgK9IHxp2aktbssMBTZUoA2P0&s=QmpnH > wqCC8ZBxRGZgfCZuF-9SielPs89Yd-k7crooC8&e= > > > > If you install the hwloc package you can use “hwloc-calc pci=0000:01:02.0 > -- > intersect Socket” to show the NUMA node that is local to a pci device. > However I have seen some systems which report that pci device is local to > both nodes. The lstopo command can also be used for this. > > > > Hope some of that helps, > > > > Billy. > > > > > > From: ovs-discuss-boun...@openvswitch.org [mailto:ovs-discuss- > boun...@openvswitch.org] On Behalf Of Darrell Ball > > Sent: Wednesday, March 29, 2017 2:36 AM > > To: Brad Cowie <b...@cowie.nz>; Shivaram Mysore > <shivaram.mys...@gmail.com> > > Cc: ovs-discuss@openvswitch.org > > Subject: Re: [ovs-discuss] Error attaching device to DPDK > > > > > > > > From: <ovs-discuss-boun...@openvswitch.org> on behalf of Brad Cowie > <b...@cowie.nz> > > Date: Tuesday, March 28, 2017 at 5:35 PM > > To: Shivaram Mysore <shivaram.mys...@gmail.com> > > Cc: "ovs-discuss@openvswitch.org" <ovs-discuss@openvswitch.org> > > Subject: Re: [ovs-discuss] Error attaching device to DPDK > > > > Hi folks, > > I have been helping Shivaram off-list with this problem and just wanted to > update this thread with the solution. > > Basically the issue was with how hugepages were being preallocated by > OVS for use with DPDK. I believe by default OVS sets other_config:dpdk- > socket-mem to "1024,0". Which will give one 1GB hugepage to CPU0 and > none to CPU1. > > The ports Shivaram was trying to assign to DPDK were connected physically > via a PCI-E bus to CPU1 and since there are no hugepages preallocated there > for DPDK then we can't add it. > > The fix is simple, to assign a hugepage to CPU1: > > > > $ sudo ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket- > mem="1024,1024" > > In OVS 2.7.0 unfortunately due to an issue (that is now fixed by commit > ef6cca6fdc67f3cee21c6bb1c13c4ca7f8241314) you get a segfault rather than > an error message telling you the problem. > > In OVS 2.7.90 however the issue becomes quite clear as there is a nice log > message like such: > > > > 2017-03-29T00:19:57Z|00067|netdev_dpdk|ERR|Insufficient memory to > create memory pool for netdev dpdk-p0, with MTU 1500 on socket 1 > > There is documentation regarding these fields, but for case, the > documentation could simply state something like > > “you need to allocate hugepage memory on each numa node where dpdk > ports are bound and or those ports > > will not work with dpdk” > > The error log could also be a little more succinct like “no memory pool > allocated for netdev dpdk-p0,.., usage on socket 1” > > > > Possibly, going even further, when a PMD thread gets auto-allocated to > run on a given numa node, > > memory allocation is defaulted there as well. If it cannot be allocated, > an > error log is generated. > > > > Unfortunately to those unfamiliar with DPDK and NUMA architectures this > won't be very obvious. Potentially we could add some additional help to the > DPDK documentation pages for OVS that explains the other_config options > you may need to reconfigure on multi-CPU NUMA machines? > > Brad > > > > On 29 March 2017 at 10:02, Shivaram Mysore > <shivaram.mys...@gmail.com> wrote: > > Ok thanks. Will remember the same. Right now, I will not specify > anything > as I have to get this to work first! > > > > On Tue, Mar 28, 2017 at 2:00 PM, Bodireddy, Bhanuprakash > <bhanuprakash.bodire...@intel.com> wrote: > > >Question: > > > > > >I don't have any specific cpu cores associated with DPDK. Will this make > any > > >difference? I also did not see this requirement as a MUST have in the > > >documentation. > > > > I assume your question is on PMD threads. If you don’t specify any thing > explicitly > > PMD pmd thread shall be created and pinned to core 0. You can explicitly > select the no. of pmd threads > > and the cores by explicitly setting the affinity mask using > 'other_config:pmd-cpu-mask'. > > > > Based on the affinity mask, the PMD threads shall be spawned and the rx > queues will be uniformly > > distributed among the PMD threads. > > > > Refer https://urldefense.proofpoint.com/v2/url?u=http- > 3A__docs.openvswitch.org_en_latest_howto_dpdk_&d=DwIGaQ&c=uilaK9 > 0D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih- > uZnsw&m=IKHiqOFj5oogxEfx6OMgK9IHxp2aktbssMBTZUoA2P0&s=JNQjCdB > cYJG8H1n4bT6k_zFhtkk6wcRE27AhEuQoqjg&e= . > > > > Also note that 'dpdk-lcore-mask' should be used to set the dpdk lcore > threads and vswitchd. > > > > - Bhanuprakash. > > > > > > > ># cat /proc/cmdline > > >BOOT_IMAGE=/boot/vmlinuz-4.10.0-14-generic > root=/dev/mapper/intelsdn- > > >-vg-root ro quiet intel_iommu=on iommu=pt default_hugepagesz=1G > > >hugepagesz=1G hugepages=4 > > > > > > > > >Thanks > > > > > >/Shivaram > > > > > >On Tue, Mar 28, 2017 at 1:32 PM, Shivaram Mysore > > ><shivaram.mys...@gmail.com> wrote: > > >I was advised on this mailing list that VFIO drivers are better than > IGB_UIO > + > > >more forgiving [1]. Hence switched to the same. > > > > > >[1] https://urldefense.proofpoint.com/v2/url?u=http-3A__dpdk- > 2Dguide.gitlab.io_dpdk- > 2Dguide_setup_binding.html&d=DwIGaQ&c=uilaK90D4TOVoH58JNXRgQ&r= > BVhFA09CGX7JQ5Ih- > uZnsw&m=IKHiqOFj5oogxEfx6OMgK9IHxp2aktbssMBTZUoA2P0&s=KgBSWU > sNwqMWB_Pm2CHmGAVAO5enRJxRlvbS9MR-7Ww&e= > > > > > >/Shivaram > > > > > >On Tue, Mar 28, 2017 at 1:12 PM, Bodireddy, Bhanuprakash > > ><bhanuprakash.bodire...@intel.com> wrote: > > > > > >>any hints for me based on the info provided? > > > > > >Your BIOS options and bootcmd line options are appropriate and no issues > > >there. I wanted to check if you tried igb_uio drivers > > >but saw your post here > https://urldefense.proofpoint.com/v2/url?u=https- > 3A__mail.openvswitch.org_pipermail_ovs- > 2D&d=DwIGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih- > uZnsw&m=IKHiqOFj5oogxEfx6OMgK9IHxp2aktbssMBTZUoA2P0&s=qoh0Qu6 > HkVOjRIO_oNqdzBGTGjRdo5tppjkG2R3KpEw&e= > > >discuss/2017-March/043993.html, reporting a crash with UIO drivers. > > > > > >Is that the reason why you moved to VFIO? I cross checked your logs but > > >found nothing suspicious there. > > > > > >-Bhanuprakash. > > > > > >> > > >>Thanks > > >> > > >>On Tue, Mar 28, 2017 at 8:41 AM, Shivaram Mysore > > >><shivaram.mys...@gmail.com> wrote: > > >>Hi, > > >> > > >># dmesg | grep -e DMAR -e IOMMU > > >>[ 0.000000] ACPI: DMAR 0x00000000798AF340 000158 (v01 SUPERM > SMCI-- > > >>MB 00000001 INTL 20091013) > > >>[ 0.000000] DMAR: IOMMU enabled > > >>[ 0.109987] DMAR: Host address width 46 > > >>[ 0.109988] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0 > > >>[ 0.109997] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap > > >>d2078c106f0466 ecap f020de > > >>[ 0.109998] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1 > > >>[ 0.110002] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap > > >>d2078c106f0466 ecap f020de > > >>[ 0.110003] DMAR: RMRR base: 0x0000007ba75000 end: > 0x0000007ba84fff > > >>[ 0.110004] DMAR: ATSR flags: 0x0 > > >>[ 0.110005] DMAR: RHSA base: 0x000000c7ffc000 proximity domain: 0x0 > > >>[ 0.110006] DMAR: RHSA base: 0x000000fbffc000 proximity domain: 0x1 > > >>[ 0.110008] DMAR-IR: IOAPIC id 3 under DRHD base 0xfbffc000 IOMMU > 0 > > >>[ 0.110008] DMAR-IR: IOAPIC id 1 under DRHD base 0xc7ffc000 IOMMU > 1 > > >>[ 0.110009] DMAR-IR: IOAPIC id 2 under DRHD base 0xc7ffc000 IOMMU > 1 > > >>[ 0.110010] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000 > > >>[ 0.110011] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt > out > > >bit. > > >>[ 0.110012] DMAR-IR: Use 'intremap=no_x2apic_optout' to override > the > > >>BIOS setting. > > >>[ 0.110902] DMAR-IR: Enabled IRQ remapping in xapic mode > > >>[ 1.687595] DMAR: dmar0: Using Queued invalidation > > >>[ 1.687608] DMAR: dmar1: Using Queued invalidation > > >>[ 1.688473] DMAR: Hardware identity mapping for device 0000:ff:08.0 > > >>[ 1.688475] DMAR: Hardware identity mapping for device 0000:ff:08.2 > > >>[ 1.688477] DMAR: Hardware identity mapping for device 0000:ff:08.3 > > >>[ 1.688478] DMAR: Hardware identity mapping for device 0000:ff:09.0 > > >>[ 1.688480] DMAR: Hardware identity mapping for device 0000:ff:09.2 > > >>[ 1.688482] DMAR: Hardware identity mapping for device 0000:ff:09.3 > > >>[ 1.688483] DMAR: Hardware identity mapping for device 0000:ff:0b.0 > > >>[ 1.688485] DMAR: Hardware identity mapping for device 0000:ff:0b.1 > > >>[ 1.688487] DMAR: Hardware identity mapping for device 0000:ff:0b.2 > > >>[ 1.688488] DMAR: Hardware identity mapping for device 0000:ff:0c.0 > > >>[ 1.688490] DMAR: Hardware identity mapping for device 0000:ff:0c.1 > > >>[ 1.688491] DMAR: Hardware identity mapping for device 0000:ff:0c.2 > > >>[ 1.688493] DMAR: Hardware identity mapping for device 0000:ff:0c.3 > > >>[ 1.688494] DMAR: Hardware identity mapping for device 0000:ff:0c.4 > > >>[ 1.688496] DMAR: Hardware identity mapping for device 0000:ff:0c.5 > > >>[ 1.688498] DMAR: Hardware identity mapping for device 0000:ff:0c.6 > > >>[ 1.688499] DMAR: Hardware identity mapping for device 0000:ff:0c.7 > > >>[ 1.688500] DMAR: Hardware identity mapping for device 0000:ff:0d.0 > > >>[ 1.688502] DMAR: Hardware identity mapping for device 0000:ff:0d.1 > > >>[ 1.688504] DMAR: Hardware identity mapping for device 0000:ff:0d.2 > > >>[ 1.688505] DMAR: Hardware identity mapping for device 0000:ff:0d.3 > > >>[ 1.688506] DMAR: Hardware identity mapping for device 0000:ff:0d.4 > > >>[ 1.688508] DMAR: Hardware identity mapping for device 0000:ff:0d.5 > > >>[ 1.688509] DMAR: Hardware identity mapping for device 0000:ff:0d.6 > > >>[ 1.688510] DMAR: Hardware identity mapping for device 0000:ff:0d.7 > > >>[ 1.688511] DMAR: Hardware identity mapping for device 0000:ff:0e.0 > > >>[ 1.688513] DMAR: Hardware identity mapping for device 0000:ff:0e.1 > > >>[ 1.688514] DMAR: Hardware identity mapping for device 0000:ff:0f.0 > > >>[ 1.688516] DMAR: Hardware identity mapping for device 0000:ff:0f.1 > > >>[ 1.688517] DMAR: Hardware identity mapping for device 0000:ff:0f.2 > > >>[ 1.688518] DMAR: Hardware identity mapping for device 0000:ff:0f.3 > > >>[ 1.688520] DMAR: Hardware identity mapping for device 0000:ff:0f.4 > > >>[ 1.688521] DMAR: Hardware identity mapping for device 0000:ff:0f.5 > > >>[ 1.688523] DMAR: Hardware identity mapping for device 0000:ff:0f.6 > > >>[ 1.688524] DMAR: Hardware identity mapping for device 0000:ff:10.0 > > >>[ 1.688526] DMAR: Hardware identity mapping for device 0000:ff:10.1 > > >>[ 1.688527] DMAR: Hardware identity mapping for device 0000:ff:10.5 > > >>[ 1.688529] DMAR: Hardware identity mapping for device 0000:ff:10.6 > > >>[ 1.688530] DMAR: Hardware identity mapping for device 0000:ff:10.7 > > >>[ 1.688532] DMAR: Hardware identity mapping for device 0000:ff:12.0 > > >>[ 1.688533] DMAR: Hardware identity mapping for device 0000:ff:12.1 > > >>[ 1.688534] DMAR: Hardware identity mapping for device 0000:ff:12.4 > > >>[ 1.688536] DMAR: Hardware identity mapping for device 0000:ff:12.5 > > >>[ 1.688537] DMAR: Hardware identity mapping for device 0000:ff:13.0 > > >>[ 1.688539] DMAR: Hardware identity mapping for device 0000:ff:13.1 > > >>[ 1.688541] DMAR: Hardware identity mapping for device 0000:ff:13.2 > > >>[ 1.688542] DMAR: Hardware identity mapping for device 0000:ff:13.3 > > >>[ 1.688543] DMAR: Hardware identity mapping for device 0000:ff:13.6 > > >>[ 1.688545] DMAR: Hardware identity mapping for device 0000:ff:13.7 > > >>[ 1.688546] DMAR: Hardware identity mapping for device 0000:ff:14.0 > > >>[ 1.688547] DMAR: Hardware identity mapping for device 0000:ff:14.1 > > >>[ 1.688549] DMAR: Hardware identity mapping for device 0000:ff:14.2 > > >>[ 1.688550] DMAR: Hardware identity mapping for device 0000:ff:14.3 > > >>[ 1.688552] DMAR: Hardware identity mapping for device 0000:ff:14.4 > > >>[ 1.688553] DMAR: Hardware identity mapping for device 0000:ff:14.5 > > >>[ 1.688559] DMAR: Hardware identity mapping for device 0000:ff:14.6 > > >>[ 1.688561] DMAR: Hardware identity mapping for device 0000:ff:14.7 > > >>[ 1.688562] DMAR: Hardware identity mapping for device 0000:ff:16.0 > > >>[ 1.688564] DMAR: Hardware identity mapping for device 0000:ff:16.1 > > >>[ 1.688565] DMAR: Hardware identity mapping for device 0000:ff:16.2 > > >>[ 1.688566] DMAR: Hardware identity mapping for device 0000:ff:16.3 > > >>[ 1.688568] DMAR: Hardware identity mapping for device 0000:ff:16.6 > > >>[ 1.688569] DMAR: Hardware identity mapping for device 0000:ff:16.7 > > >>[ 1.688571] DMAR: Hardware identity mapping for device 0000:ff:17.0 > > >>[ 1.688573] DMAR: Hardware identity mapping for device 0000:ff:17.1 > > >>[ 1.688574] DMAR: Hardware identity mapping for device 0000:ff:17.2 > > >>[ 1.688576] DMAR: Hardware identity mapping for device 0000:ff:17.3 > > >>[ 1.688577] DMAR: Hardware identity mapping for device 0000:ff:17.4 > > >>[ 1.688579] DMAR: Hardware identity mapping for device 0000:ff:17.5 > > >>[ 1.688581] DMAR: Hardware identity mapping for device 0000:ff:17.6 > > >>[ 1.688582] DMAR: Hardware identity mapping for device 0000:ff:17.7 > > >>[ 1.688584] DMAR: Hardware identity mapping for device 0000:ff:1e.0 > > >>[ 1.688586] DMAR: Hardware identity mapping for device 0000:ff:1e.1 > > >>[ 1.688588] DMAR: Hardware identity mapping for device 0000:ff:1e.2 > > >>[ 1.688590] DMAR: Hardware identity mapping for device 0000:ff:1e.3 > > >>[ 1.688592] DMAR: Hardware identity mapping for device 0000:ff:1e.4 > > >>[ 1.688594] DMAR: Hardware identity mapping for device 0000:ff:1f.0 > > >>[ 1.688596] DMAR: Hardware identity mapping for device 0000:ff:1f.2 > > >>[ 1.688610] DMAR: Hardware identity mapping for device 0000:7f:08.0 > > >>[ 1.688612] DMAR: Hardware identity mapping for device 0000:7f:08.2 > > >>[ 1.688614] DMAR: Hardware identity mapping for device 0000:7f:08.3 > > >>[ 1.688616] DMAR: Hardware identity mapping for device 0000:7f:09.0 > > >>[ 1.688617] DMAR: Hardware identity mapping for device 0000:7f:09.2 > > >>[ 1.688619] DMAR: Hardware identity mapping for device 0000:7f:09.3 > > >>[ 1.688621] DMAR: Hardware identity mapping for device 0000:7f:0b.0 > > >>[ 1.688623] DMAR: Hardware identity mapping for device 0000:7f:0b.1 > > >>[ 1.688625] DMAR: Hardware identity mapping for device 0000:7f:0b.2 > > >>[ 1.688627] DMAR: Hardware identity mapping for device 0000:7f:0c.0 > > >>[ 1.688629] DMAR: Hardware identity mapping for device 0000:7f:0c.1 > > >>[ 1.688631] DMAR: Hardware identity mapping for device 0000:7f:0c.2 > > >>[ 1.688632] DMAR: Hardware identity mapping for device 0000:7f:0c.3 > > >>[ 1.688634] DMAR: Hardware identity mapping for device 0000:7f:0c.4 > > >>[ 1.688636] DMAR: Hardware identity mapping for device 0000:7f:0c.5 > > >>[ 1.688637] DMAR: Hardware identity mapping for device 0000:7f:0c.6 > > >>[ 1.688639] DMAR: Hardware identity mapping for device 0000:7f:0c.7 > > >>[ 1.688641] DMAR: Hardware identity mapping for device 0000:7f:0d.0 > > >>[ 1.688642] DMAR: Hardware identity mapping for device 0000:7f:0d.1 > > >>[ 1.688644] DMAR: Hardware identity mapping for device 0000:7f:0d.2 > > >>[ 1.688646] DMAR: Hardware identity mapping for device 0000:7f:0d.3 > > >>[ 1.688647] DMAR: Hardware identity mapping for device 0000:7f:0d.4 > > >>[ 1.688649] DMAR: Hardware identity mapping for device 0000:7f:0d.5 > > >>[ 1.688650] DMAR: Hardware identity mapping for device 0000:7f:0d.6 > > >>[ 1.688652] DMAR: Hardware identity mapping for device 0000:7f:0d.7 > > >>[ 1.688653] DMAR: Hardware identity mapping for device 0000:7f:0e.0 > > >>[ 1.688655] DMAR: Hardware identity mapping for device 0000:7f:0e.1 > > >>[ 1.688657] DMAR: Hardware identity mapping for device 0000:7f:0f.0 > > >>[ 1.688658] DMAR: Hardware identity mapping for device 0000:7f:0f.1 > > >>[ 1.688660] DMAR: Hardware identity mapping for device 0000:7f:0f.2 > > >>[ 1.688661] DMAR: Hardware identity mapping for device 0000:7f:0f.3 > > >>[ 1.688663] DMAR: Hardware identity mapping for device 0000:7f:0f.4 > > >>[ 1.688664] DMAR: Hardware identity mapping for device 0000:7f:0f.5 > > >>[ 1.688666] DMAR: Hardware identity mapping for device 0000:7f:0f.6 > > >>[ 1.688668] DMAR: Hardware identity mapping for device 0000:7f:10.0 > > >>[ 1.688669] DMAR: Hardware identity mapping for device 0000:7f:10.1 > > >>[ 1.688671] DMAR: Hardware identity mapping for device 0000:7f:10.5 > > >>[ 1.688673] DMAR: Hardware identity mapping for device 0000:7f:10.6 > > >>[ 1.688674] DMAR: Hardware identity mapping for device 0000:7f:10.7 > > >>[ 1.688676] DMAR: Hardware identity mapping for device 0000:7f:12.0 > > >>[ 1.688678] DMAR: Hardware identity mapping for device 0000:7f:12.1 > > >>[ 1.688684] DMAR: Hardware identity mapping for device 0000:7f:12.4 > > >>[ 1.688686] DMAR: Hardware identity mapping for device 0000:7f:12.5 > > >>[ 1.688688] DMAR: Hardware identity mapping for device 0000:7f:13.0 > > >>[ 1.688689] DMAR: Hardware identity mapping for device 0000:7f:13.1 > > >>[ 1.688691] DMAR: Hardware identity mapping for device 0000:7f:13.2 > > >>[ 1.688692] DMAR: Hardware identity mapping for device 0000:7f:13.3 > > >>[ 1.688694] DMAR: Hardware identity mapping for device 0000:7f:13.6 > > >>[ 1.688696] DMAR: Hardware identity mapping for device 0000:7f:13.7 > > >>[ 1.688697] DMAR: Hardware identity mapping for device 0000:7f:14.0 > > >>[ 1.688699] DMAR: Hardware identity mapping for device 0000:7f:14.1 > > >>[ 1.688701] DMAR: Hardware identity mapping for device 0000:7f:14.2 > > >>[ 1.688703] DMAR: Hardware identity mapping for device 0000:7f:14.3 > > >>[ 1.688704] DMAR: Hardware identity mapping for device 0000:7f:14.4 > > >>[ 1.688706] DMAR: Hardware identity mapping for device 0000:7f:14.5 > > >>[ 1.688708] DMAR: Hardware identity mapping for device 0000:7f:14.6 > > >>[ 1.688710] DMAR: Hardware identity mapping for device 0000:7f:14.7 > > >>[ 1.688712] DMAR: Hardware identity mapping for device 0000:7f:16.0 > > >>[ 1.688714] DMAR: Hardware identity mapping for device 0000:7f:16.1 > > >>[ 1.688716] DMAR: Hardware identity mapping for device 0000:7f:16.2 > > >>[ 1.688718] DMAR: Hardware identity mapping for device 0000:7f:16.3 > > >>[ 1.688720] DMAR: Hardware identity mapping for device 0000:7f:16.6 > > >>[ 1.688722] DMAR: Hardware identity mapping for device 0000:7f:16.7 > > >>[ 1.688724] DMAR: Hardware identity mapping for device 0000:7f:17.0 > > >>[ 1.688726] DMAR: Hardware identity mapping for device 0000:7f:17.1 > > >>[ 1.688729] DMAR: Hardware identity mapping for device 0000:7f:17.2 > > >>[ 1.688731] DMAR: Hardware identity mapping for device 0000:7f:17.3 > > >>[ 1.688733] DMAR: Hardware identity mapping for device 0000:7f:17.4 > > >>[ 1.688735] DMAR: Hardware identity mapping for device 0000:7f:17.5 > > >>[ 1.688737] DMAR: Hardware identity mapping for device 0000:7f:17.6 > > >>[ 1.688739] DMAR: Hardware identity mapping for device 0000:7f:17.7 > > >>[ 1.688741] DMAR: Hardware identity mapping for device 0000:7f:1e.0 > > >>[ 1.688742] DMAR: Hardware identity mapping for device 0000:7f:1e.1 > > >>[ 1.688744] DMAR: Hardware identity mapping for device 0000:7f:1e.2 > > >>[ 1.688746] DMAR: Hardware identity mapping for device 0000:7f:1e.3 > > >>[ 1.688748] DMAR: Hardware identity mapping for device 0000:7f:1e.4 > > >>[ 1.688750] DMAR: Hardware identity mapping for device 0000:7f:1f.0 > > >>[ 1.688752] DMAR: Hardware identity mapping for device 0000:7f:1f.2 > > >>[ 1.688769] DMAR: Hardware identity mapping for device 0000:00:00.0 > > >>[ 1.688774] DMAR: Hardware identity mapping for device 0000:00:01.0 > > >>[ 1.688778] DMAR: Hardware identity mapping for device 0000:00:01.1 > > >>[ 1.688783] DMAR: Hardware identity mapping for device 0000:00:03.0 > > >>[ 1.688785] DMAR: Hardware identity mapping for device 0000:00:04.0 > > >>[ 1.688787] DMAR: Hardware identity mapping for device 0000:00:04.1 > > >>[ 1.688790] DMAR: Hardware identity mapping for device 0000:00:04.2 > > >>[ 1.688792] DMAR: Hardware identity mapping for device 0000:00:04.3 > > >>[ 1.688794] DMAR: Hardware identity mapping for device 0000:00:04.4 > > >>[ 1.688797] DMAR: Hardware identity mapping for device 0000:00:04.5 > > >>[ 1.688799] DMAR: Hardware identity mapping for device 0000:00:04.6 > > >>[ 1.688801] DMAR: Hardware identity mapping for device 0000:00:04.7 > > >>[ 1.688803] DMAR: Hardware identity mapping for device 0000:00:05.0 > > >>[ 1.688807] DMAR: Hardware identity mapping for device 0000:00:05.1 > > >>[ 1.688809] DMAR: Hardware identity mapping for device 0000:00:05.2 > > >>[ 1.688811] DMAR: Hardware identity mapping for device 0000:00:05.4 > > >>[ 1.688814] DMAR: Hardware identity mapping for device 0000:00:11.0 > > >>[ 1.688816] DMAR: Hardware identity mapping for device 0000:00:11.4 > > >>[ 1.688818] DMAR: Hardware identity mapping for device 0000:00:14.0 > > >>[ 1.688821] DMAR: Hardware identity mapping for device 0000:00:16.0 > > >>[ 1.688823] DMAR: Hardware identity mapping for device 0000:00:16.1 > > >>[ 1.688825] DMAR: Hardware identity mapping for device 0000:00:1a.0 > > >>[ 1.688828] DMAR: Hardware identity mapping for device 0000:00:1c.0 > > >>[ 1.688831] DMAR: Hardware identity mapping for device 0000:00:1c.2 > > >>[ 1.688833] DMAR: Hardware identity mapping for device 0000:00:1d.0 > > >>[ 1.688835] DMAR: Hardware identity mapping for device 0000:00:1f.0 > > >>[ 1.688837] DMAR: Hardware identity mapping for device 0000:00:1f.2 > > >>[ 1.688844] DMAR: Hardware identity mapping for device 0000:00:1f.3 > > >>[ 1.688846] DMAR: Hardware identity mapping for device 0000:00:1f.6 > > >>[ 1.688864] DMAR: Hardware identity mapping for device 0000:01:00.0 > > >>[ 1.688870] DMAR: Hardware identity mapping for device 0000:01:00.1 > > >>[ 1.688876] DMAR: Hardware identity mapping for device 0000:01:00.2 > > >>[ 1.688883] DMAR: Hardware identity mapping for device 0000:01:00.3 > > >>[ 1.688895] DMAR: Hardware identity mapping for device 0000:80:01.0 > > >>[ 1.688899] DMAR: Hardware identity mapping for device 0000:80:02.0 > > >>[ 1.688903] DMAR: Hardware identity mapping for device 0000:80:03.0 > > >>[ 1.688906] DMAR: Hardware identity mapping for device 0000:80:04.0 > > >>[ 1.688908] DMAR: Hardware identity mapping for device 0000:80:04.1 > > >>[ 1.688911] DMAR: Hardware identity mapping for device 0000:80:04.2 > > >>[ 1.688913] DMAR: Hardware identity mapping for device 0000:80:04.3 > > >>[ 1.688915] DMAR: Hardware identity mapping for device 0000:80:04.4 > > >>[ 1.688918] DMAR: Hardware identity mapping for device 0000:80:04.5 > > >>[ 1.688920] DMAR: Hardware identity mapping for device 0000:80:04.6 > > >>[ 1.688923] DMAR: Hardware identity mapping for device 0000:80:04.7 > > >>[ 1.688936] DMAR: Hardware identity mapping for device 0000:80:05.0 > > >>[ 1.688940] DMAR: Hardware identity mapping for device 0000:80:05.1 > > >>[ 1.688943] DMAR: Hardware identity mapping for device 0000:80:05.2 > > >>[ 1.688945] DMAR: Hardware identity mapping for device 0000:80:05.4 > > >>[ 1.688956] DMAR: Hardware identity mapping for device 0000:81:00.0 > > >>[ 1.688961] DMAR: Hardware identity mapping for device 0000:81:00.1 > > >>[ 1.688974] DMAR: Hardware identity mapping for device 0000:82:00.0 > > >>[ 1.688980] DMAR: Hardware identity mapping for device 0000:82:00.1 > > >>[ 1.688986] DMAR: Hardware identity mapping for device 0000:82:00.2 > > >>[ 1.688992] DMAR: Hardware identity mapping for device 0000:82:00.3 > > >>[ 1.689006] DMAR: Hardware identity mapping for device 0000:83:00.0 > > >>[ 1.689012] DMAR: Hardware identity mapping for device 0000:83:00.1 > > >>[ 1.689018] DMAR: Hardware identity mapping for device 0000:83:00.2 > > >>[ 1.689024] DMAR: Hardware identity mapping for device 0000:83:00.3 > > >>[ 1.689025] DMAR: Setting RMRR: > > >>[ 1.689026] DMAR: Ignoring identity map for HW passthrough device > > >>0000:00:14.0 [0x7ba75000 - 0x7ba84fff] > > >>[ 1.689027] DMAR: Ignoring identity map for HW passthrough device > > >>0000:00:1a.0 [0x7ba75000 - 0x7ba84fff] > > >>[ 1.689028] DMAR: Ignoring identity map for HW passthrough device > > >>0000:00:1d.0 [0x7ba75000 - 0x7ba84fff] > > >>[ 1.689036] DMAR: Prepare 0-16MiB unity mapping for LPC > > >>[ 1.689036] DMAR: Ignoring identity map for HW passthrough device > > >>0000:00:1f.0 [0x0 - 0xffffff] > > >>[ 1.689040] DMAR: Intel(R) Virtualization Technology for Directed I/O > > >>[ 2.299766] DMAR: 32bit 0000:00:11.4 uses non-identity mapping > > >>[ 2.301941] DMAR: 32bit 0000:00:1a.0 uses non-identity mapping > > >>[ 2.302679] DMAR: Setting identity map for device 0000:00:1a.0 > [0x7ba75000 > > >- > > >>0x7ba84fff] > > >>[ 2.320910] DMAR: 32bit 0000:00:1d.0 uses non-identity mapping > > >>[ 2.321685] DMAR: Setting identity map for device 0000:00:1d.0 > > >[0x7ba75000 > > >>- 0x7ba84fff] > > >># > > >> > > >> > > >>On Tue, Mar 28, 2017 at 1:46 AM, Bodireddy, Bhanuprakash > > >><bhanuprakash.bodire...@intel.com> wrote: > > >>Hello shivaram, > > >> > > >>Darrell has already asked if VT-d is enabled? > > >>Your cmdline options are appropriate as you have iommu > > >>enabled(intel_iommu=on iommu=pt). > > >>can you also check the output of below mentioned command to see if > VT-d > > >is > > >>enabled in BIOS? > > >> > > >>$ dmesg | grep -e DMAR -e IOMMU > > >> > > >>Regards, > > >>Bhanuprakash. > > >> > > >>From: ovs-discuss-boun...@openvswitch.org [mailto:ovs-discuss- > > >>boun...@openvswitch.org] On Behalf Of Shivaram Mysore > > >>Sent: Tuesday, March 28, 2017 6:25 AM > > >>To: Darrell Ball <db...@vmware.com> > > >>Cc: ovs-discuss@openvswitch.org > > >>Subject: Re: [ovs-discuss] Error attaching device to DPDK > > >> > > >># uname -r > > >>4.10.0-14-generic > > >> > > >># cat /proc/cmdline > > >>BOOT_IMAGE=/boot/vmlinuz-4.10.0-14-generic > > >root=/dev/mapper/intelsdn- > > >>-vg-root ro quiet intel_iommu=on iommu=pt default_hugepagesz=1G > > >>hugepagesz=1G hugepages=4 > > >> > > >># uname -a > > >>Linux intelsdn 4.10.0-14-generic #16-Ubuntu SMP Fri Mar 17 15:19:26 > UTC > > >2017 > > >>x86_64 x86_64 x86_64 GNU/Linux > > >> > > >># cat /etc/*release > > >>DISTRIB_ID=Ubuntu > > >>DISTRIB_RELEASE=17.04 > > >>DISTRIB_CODENAME=zesty > > >>DISTRIB_DESCRIPTION="Ubuntu Zesty Zapus (development branch)" > > >>NAME="Ubuntu" > > >>VERSION="17.04 (Zesty Zapus)" > > >>ID=ubuntu > > >>ID_LIKE=debian > > >>PRETTY_NAME="Ubuntu Zesty Zapus (development branch)" > > >>VERSION_ID="17.04" > > >> > > >> > > >>Nothing in the syslog related > > >> > > >>On Mon, Mar 27, 2017 at 10:02 PM, Darrell Ball <db...@vmware.com> > wrote: > > >>I see the pastebins now. > > >> > > >>What kernel version are you using ? > > >>VT-d enabled ? > > >>Anything related in syslog ? > > >> > > >>From: Shivaram Mysore <shivaram.mys...@gmail.com> > > >>Date: Monday, March 27, 2017 at 9:34 PM > > >>To: Darrell Ball <db...@vmware.com> > > >>Cc: "ovs-discuss@openvswitch.org" <ovs-discuss@openvswitch.org> > > >>Subject: Re: [ovs-discuss] Error attaching device to DPDK > > >> > > >>One more datapoint, please check the paste bin log in URL sent > > >earlier. Dpdk- > > >>devbind works and you can see results in the paste bin log > > >> > > >>/Shivaram > > >>::Sent from my mobile device:: > > >> > > >>On Mar 27, 2017, at 8:34 PM, Darrell Ball <db...@vmware.com> wrote: > > >>see inline below > > >> > > >>But a few questions: > > >>1) Are you using the recommended version of dpdk with OVS 2.7 – > 16.11 ? > > >>2) Are you getting a crash still or is this post crash ? > > >>3) Was the previous crash still happening with 16.11 or only 17.02 > ? > > >> > > >>Thanks > > >>Darrell > > >> > > >>From: <ovs-discuss-boun...@openvswitch.org> on behalf of Shivaram > > >>Mysore <shivaram.mys...@gmail.com> > > >>Date: Monday, March 27, 2017 at 7:45 PM > > >>To: "ovs-discuss@openvswitch.org" <ovs-discuss@openvswitch.org> > > >>Subject: [ovs-discuss] Error attaching device to DPDK > > >> > > >>Hello, > > >> > > >>I have installed OVS 2.7 running. When I do: > > >>1. # ovs-vsctl add-port ovs-ip64-br0 dpdk-p0 -- set Interface > dpdk-p0 > > >>type=dpdk options:dpdk-devargs=0000:82:00.0 > > >>2. ovs-vsctl: Error detected while setting up 'dpdk-p0': Error > attaching > > >>device '0000:82:00.0' to DPDK. See ovs-vswitchd log for details. > > >> > > >>Possibly, the port is not bound. > > >>Check using > > >>$DPDK_DIR/tools/dpdk-devbind.py –status > > >>and see the documentation for more detail > > >> > > >>and the log file shows: > > >>1. 2017-03-28T02:04:40.004Z|00064|netdev_dpdk|WARN|Error > attaching > > >>device '0000:82:00.0' to DPDK > > >>2. 2017-03-28T02:04:40.004Z|00065|netdev|WARN|dpdk-p0: could > > >>not set configuration (Invalid argument) > > >> > > >>What would cause such a situation? > > >> > > >>Commands run: https://urldefense.proofpoint.com/v2/url?u=https- > 3A__pastebin.com_0P754x0t&d=DwIGaQ&c=uilaK90D4TOVoH58JNXRgQ&r= > BVhFA09CGX7JQ5Ih- > uZnsw&m=IKHiqOFj5oogxEfx6OMgK9IHxp2aktbssMBTZUoA2P0&s=XEhP0isS > M6FAfGw-HXbOfMLfomYnhSo1G2UxvC1qjp8&e= > > >>Detailed OVS Log: https://urldefense.proofpoint.com/v2/url?u=https- > 3A__pastebin.com_ctBhLuRG&d=DwIGaQ&c=uilaK90D4TOVoH58JNXRgQ&r= > BVhFA09CGX7JQ5Ih- > uZnsw&m=IKHiqOFj5oogxEfx6OMgK9IHxp2aktbssMBTZUoA2P0&s=90tudSR > NE_HlxrCclX83wTmqlUL5xMPrRha7_-fMAKs&e= > > >> > > >> > > > > > > > > > _______________________________________________ > > discuss mailing list > > disc...@openvswitch.org > > https://urldefense.proofpoint.com/v2/url?u=https- > 3A__mail.openvswitch.org_mailman_listinfo_ovs- > 2Ddiscuss&d=DwIGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih > -uZnsw&m=IKHiqOFj5oogxEfx6OMgK9IHxp2aktbssMBTZUoA2P0&s=9- > oI5HFUKs-b_kDAFcnn2vFwTcuE2xw2SFvd-zE6vWY&e= > > > > > > _______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
