Hi Shivram, Apologies if I missed this info in the email thread.
For the bug report: * What version of DPDK were you using? * Also what is your hardware - NICs (out from dpdk_devbind.py --status) and processor architecture (last few lines from cpuid -1 ) Thanks, Billy. > -----Original Message----- > From: O Mahony, Billy > Sent: Tuesday, April 4, 2017 12:00 PM > To: 'Darrell Ball' <db...@vmware.com>; Brad Cowie <b...@cowie.nz>; > Shivaram Mysore <shivaram.mys...@gmail.com> > Cc: ovs-discuss@openvswitch.org > Subject: RE: [ovs-discuss] Error attaching device to DPDK > > Hi Darrell, > > /Billy > > > -----Original Message----- > > From: Darrell Ball [mailto:db...@vmware.com] > > Sent: Thursday, March 30, 2017 8:51 PM > > To: O Mahony, Billy <billy.o.mah...@intel.com>; Brad Cowie > > <b...@cowie.nz>; Shivaram Mysore <shivaram.mys...@gmail.com> > > Cc: ovs-discuss@openvswitch.org > > Subject: Re: [ovs-discuss] Error attaching device to DPDK > > > > > > > > On 3/29/17, 7:40 AM, "O Mahony, Billy" <billy.o.mah...@intel.com> > wrote: > > > > Hi All, > > > > > > > > Just to add something to this conversation that has not been > > explicitly mentioned below. > > > > > > > > Brad outlines how to set other_config:dpdk-socket-mem to reserve > > hugepages from both NUMA nodes in order to be sure to avoid pci > > topology issues. > > > > > > > > Strictly this doesn’t have to be done on both nodes – just on the > > correct node :) – but sometimes it’s easier to just cover all the bases. > > > > > > > > In addition you will need to ensure that: > > > > * That linux has hugepages available on both (or the right) nodes. > > > > * There is a pmd thread created on each node by setting > > other_config:pmd-cpu-mask correctly – or again at least on the right node. > > > > > > > > These config items are described in > > > > * ./Documentation/intro/install/dpdk.rst > > > > * ./Documentation/howto/dpdk.rst > > > > > > > > Currently if there is not a pmd thread on the NUMA node belonging > > to the PCI device the device will not be polled (a warning message is > > issued to this effect). > > > > > > [Darrell] > > From Linux POV, the “default” memory policy is to allocate the > > hugepage pool across numa nodes, if possible; this assumes an > > allocation was done, which we have no reports here otherwise. > > iirc based on the logs in this case, which I cannot view anymore, > > there was no pmd mask configuration specified. > > The advertised behavior should be, in this case, that a pmd thread is > > allocated on each numa node with at least one local dpdk interface bound. > > iirc, the dpdk socket memory was left at default, which only > > pre-allocated from the hugepage pool on node 0 So, does it make sense > > to also try to pre- allocate hugepages on a numa node, where a PMD > thread is allocated ? > > Pls correct me in any respects. > > > [[BO'M]] > > What you're looking for here is something that will work (even if not > optimally). And leave the optimization of exact configurations of > pmds/hugepage-allocations and so on until later? > > So even if pmd-cpu-mask and dpdk-socket-mem other_config items are > omitted OVS will 'do the right thing' ? ie. place a pmd on each numa node and > take some hugepages on each NUMA node. > > It's hard to argue with that and I can put that on our internal backlog of > tasks. > > How about these rules? > * If pmd-cpu-mask is not set OVS creates one pmd thread for each NUMA > node. (current case) > * if dpdk-socket-mem is not set then for each NUMA node dpdk will reserve: > ** one quarter of the available hugepage memory > ** 512MB of 2MB hugepages > If hugepage granularity requires more memory may be reserved. > > > > > > > > > > > > > > There is currently a patch under consideration so that if a > > NUMA-local pmd is not available for a device then device will be > > assigned to some other pmd thread. While this is not as efficient due > > to the need for the data to travel between NUMA nodes it is less > frustrating to configure. > > > > > > > > https://urldefense.proofpoint.com/v2/url?u=https- > > 3A__mail.openvswitch.org_pipermail_ovs-2Ddev_2017- > > > 2DFebruary_329175.html&d=DwIGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=BVh > > FA09CGX7JQ5Ih- > > > uZnsw&m=IKHiqOFj5oogxEfx6OMgK9IHxp2aktbssMBTZUoA2P0&s=QmpnH > > wqCC8ZBxRGZgfCZuF-9SielPs89Yd-k7crooC8&e= > > > > > > > > If you install the hwloc package you can use “hwloc-calc > > pci=0000:01:02.0 -- intersect Socket” to show the NUMA node that is local > to a pci device. > > However I have seen some systems which report that pci device is local > > to both nodes. The lstopo command can also be used for this. > > > > > > > > Hope some of that helps, > > > > > > > > Billy. > > > > > > > > > > > > From: ovs-discuss-boun...@openvswitch.org [mailto:ovs-discuss- > > boun...@openvswitch.org] On Behalf Of Darrell Ball > > > > Sent: Wednesday, March 29, 2017 2:36 AM > > > > To: Brad Cowie <b...@cowie.nz>; Shivaram Mysore > > <shivaram.mys...@gmail.com> > > > > Cc: ovs-discuss@openvswitch.org > > > > Subject: Re: [ovs-discuss] Error attaching device to DPDK > > > > > > > > > > > > > > > > From: <ovs-discuss-boun...@openvswitch.org> on behalf of Brad > > Cowie <b...@cowie.nz> > > > > Date: Tuesday, March 28, 2017 at 5:35 PM > > > > To: Shivaram Mysore <shivaram.mys...@gmail.com> > > > > Cc: "ovs-discuss@openvswitch.org" <ovs-discuss@openvswitch.org> > > > > Subject: Re: [ovs-discuss] Error attaching device to DPDK > > > > > > > > Hi folks, > > > > I have been helping Shivaram off-list with this problem and just > > wanted to update this thread with the solution. > > > > Basically the issue was with how hugepages were being preallocated > > by OVS for use with DPDK. I believe by default OVS sets > > other_config:dpdk- socket-mem to "1024,0". Which will give one 1GB > > hugepage to CPU0 and none to CPU1. > > > > The ports Shivaram was trying to assign to DPDK were connected > > physically via a PCI-E bus to CPU1 and since there are no hugepages > > preallocated there for DPDK then we can't add it. > > > > The fix is simple, to assign a hugepage to CPU1: > > > > > > > > $ sudo ovs-vsctl --no-wait set Open_vSwitch . > > other_config:dpdk-socket- mem="1024,1024" > > > > In OVS 2.7.0 unfortunately due to an issue (that is now fixed by > > commit > > ef6cca6fdc67f3cee21c6bb1c13c4ca7f8241314) you get a segfault rather > > than an error message telling you the problem. > > > > In OVS 2.7.90 however the issue becomes quite clear as there is a > > nice log message like such: > > > > > > > > 2017-03-29T00:19:57Z|00067|netdev_dpdk|ERR|Insufficient memory to > > create memory pool for netdev dpdk-p0, with MTU 1500 on socket 1 > > > > There is documentation regarding these fields, but for case, the > > documentation could simply state something like > > > > “you need to allocate hugepage memory on each numa node where > dpdk > > ports are bound and or those ports > > > > will not work with dpdk” > > > > The error log could also be a little more succinct like “no memory > > pool allocated for netdev dpdk-p0,.., usage on socket 1” > > > > > > > > Possibly, going even further, when a PMD thread gets > > auto-allocated to run on a given numa node, > > > > memory allocation is defaulted there as well. If it cannot be > > allocated, an error log is generated. > > > > > > > > Unfortunately to those unfamiliar with DPDK and NUMA architectures > > this won't be very obvious. Potentially we could add some additional > > help to the DPDK documentation pages for OVS that explains the > > other_config options you may need to reconfigure on multi-CPU NUMA > machines? > > > > Brad > > > > > > > > On 29 March 2017 at 10:02, Shivaram Mysore > > <shivaram.mys...@gmail.com> wrote: > > > > Ok thanks. Will remember the same. Right now, I will not specify > > anything as I have to get this to work first! > > > > > > > > On Tue, Mar 28, 2017 at 2:00 PM, Bodireddy, Bhanuprakash > > <bhanuprakash.bodire...@intel.com> wrote: > > > > >Question: > > > > > > > > > >I don't have any specific cpu cores associated with DPDK. Will > > this make any > > > > >difference? I also did not see this requirement as a MUST have > > in the > > > > >documentation. > > > > > > > > I assume your question is on PMD threads. If you don’t specify any > > thing explicitly > > > > PMD pmd thread shall be created and pinned to core 0. You can > > explicitly select the no. of pmd threads > > > > and the cores by explicitly setting the affinity mask using > > 'other_config:pmd-cpu-mask'. > > > > > > > > Based on the affinity mask, the PMD threads shall be spawned and > > the rx queues will be uniformly > > > > distributed among the PMD threads. > > > > > > > > Refer https://urldefense.proofpoint.com/v2/url?u=http- > > > 3A__docs.openvswitch.org_en_latest_howto_dpdk_&d=DwIGaQ&c=uilaK9 > > 0D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih- > > > uZnsw&m=IKHiqOFj5oogxEfx6OMgK9IHxp2aktbssMBTZUoA2P0&s=JNQjCdB > > cYJG8H1n4bT6k_zFhtkk6wcRE27AhEuQoqjg&e= . > > > > > > > > Also note that 'dpdk-lcore-mask' should be used to set the dpdk > > lcore threads and vswitchd. > > > > > > > > - Bhanuprakash. > > > > > > > > > > > > > ># cat /proc/cmdline > > > > >BOOT_IMAGE=/boot/vmlinuz-4.10.0-14-generic > > root=/dev/mapper/intelsdn- > > > > >-vg-root ro quiet intel_iommu=on iommu=pt default_hugepagesz=1G > > > > >hugepagesz=1G hugepages=4 > > > > > > > > > > > > > > >Thanks > > > > > > > > > >/Shivaram > > > > > > > > > >On Tue, Mar 28, 2017 at 1:32 PM, Shivaram Mysore > > > > ><shivaram.mys...@gmail.com> wrote: > > > > >I was advised on this mailing list that VFIO drivers are better > > than IGB_UIO > > + > > > > >more forgiving [1]. Hence switched to the same. > > > > > > > > > >[1] https://urldefense.proofpoint.com/v2/url?u=http-3A__dpdk- > > 2Dguide.gitlab.io_dpdk- > > > 2Dguide_setup_binding.html&d=DwIGaQ&c=uilaK90D4TOVoH58JNXRgQ&r= > > BVhFA09CGX7JQ5Ih- > > > uZnsw&m=IKHiqOFj5oogxEfx6OMgK9IHxp2aktbssMBTZUoA2P0&s=KgBSWU > > sNwqMWB_Pm2CHmGAVAO5enRJxRlvbS9MR-7Ww&e= > > > > > > > > > >/Shivaram > > > > > > > > > >On Tue, Mar 28, 2017 at 1:12 PM, Bodireddy, Bhanuprakash > > > > ><bhanuprakash.bodire...@intel.com> wrote: > > > > > > > > > >>any hints for me based on the info provided? > > > > > > > > > >Your BIOS options and bootcmd line options are appropriate and no > > issues > > > > >there. I wanted to check if you tried igb_uio drivers > > > > >but saw your post here > > https://urldefense.proofpoint.com/v2/url?u=https- > > 3A__mail.openvswitch.org_pipermail_ovs- > > 2D&d=DwIGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih- > > > uZnsw&m=IKHiqOFj5oogxEfx6OMgK9IHxp2aktbssMBTZUoA2P0&s=qoh0Qu6 > > HkVOjRIO_oNqdzBGTGjRdo5tppjkG2R3KpEw&e= > > > > >discuss/2017-March/043993.html, reporting a crash with UIO drivers. > > > > > > > > > >Is that the reason why you moved to VFIO? I cross checked your > > logs but > > > > >found nothing suspicious there. > > > > > > > > > >-Bhanuprakash. > > > > > > > > > >> > > > > >>Thanks > > > > >> > > > > >>On Tue, Mar 28, 2017 at 8:41 AM, Shivaram Mysore > > > > >><shivaram.mys...@gmail.com> wrote: > > > > >>Hi, > > > > >> > > > > >># dmesg | grep -e DMAR -e IOMMU > > > > >>[ 0.000000] ACPI: DMAR 0x00000000798AF340 000158 (v01 SUPERM > > SMCI-- > > > > >>MB 00000001 INTL 20091013) > > > > >>[ 0.000000] DMAR: IOMMU enabled > > > > >>[ 0.109987] DMAR: Host address width 46 > > > > >>[ 0.109988] DMAR: DRHD base: 0x000000fbffc000 flags: 0x0 > > > > >>[ 0.109997] DMAR: dmar0: reg_base_addr fbffc000 ver 1:0 cap > > > > >>d2078c106f0466 ecap f020de > > > > >>[ 0.109998] DMAR: DRHD base: 0x000000c7ffc000 flags: 0x1 > > > > >>[ 0.110002] DMAR: dmar1: reg_base_addr c7ffc000 ver 1:0 cap > > > > >>d2078c106f0466 ecap f020de > > > > >>[ 0.110003] DMAR: RMRR base: 0x0000007ba75000 end: > > 0x0000007ba84fff > > > > >>[ 0.110004] DMAR: ATSR flags: 0x0 > > > > >>[ 0.110005] DMAR: RHSA base: 0x000000c7ffc000 proximity domain: > 0x0 > > > > >>[ 0.110006] DMAR: RHSA base: 0x000000fbffc000 proximity domain: > 0x1 > > > > >>[ 0.110008] DMAR-IR: IOAPIC id 3 under DRHD base 0xfbffc000 > IOMMU > > 0 > > > > >>[ 0.110008] DMAR-IR: IOAPIC id 1 under DRHD base 0xc7ffc000 > IOMMU > > 1 > > > > >>[ 0.110009] DMAR-IR: IOAPIC id 2 under DRHD base 0xc7ffc000 > IOMMU > > 1 > > > > >>[ 0.110010] DMAR-IR: HPET id 0 under DRHD base 0xc7ffc000 > > > > >>[ 0.110011] DMAR-IR: x2apic is disabled because BIOS sets x2apic > > opt > > out > > > > >bit. > > > > >>[ 0.110012] DMAR-IR: Use 'intremap=no_x2apic_optout' to override > > the > > > > >>BIOS setting. > > > > >>[ 0.110902] DMAR-IR: Enabled IRQ remapping in xapic mode > > > > >>[ 1.687595] DMAR: dmar0: Using Queued invalidation > > > > >>[ 1.687608] DMAR: dmar1: Using Queued invalidation > > > > >>[ 1.688473] DMAR: Hardware identity mapping for device 0000:ff:08.0 > > > > >>[ 1.688475] DMAR: Hardware identity mapping for device 0000:ff:08.2 > > > > >>[ 1.688477] DMAR: Hardware identity mapping for device 0000:ff:08.3 > > > > >>[ 1.688478] DMAR: Hardware identity mapping for device 0000:ff:09.0 > > > > >>[ 1.688480] DMAR: Hardware identity mapping for device 0000:ff:09.2 > > > > >>[ 1.688482] DMAR: Hardware identity mapping for device 0000:ff:09.3 > > > > >>[ 1.688483] DMAR: Hardware identity mapping for device 0000:ff:0b.0 > > > > >>[ 1.688485] DMAR: Hardware identity mapping for device 0000:ff:0b.1 > > > > >>[ 1.688487] DMAR: Hardware identity mapping for device 0000:ff:0b.2 > > > > >>[ 1.688488] DMAR: Hardware identity mapping for device 0000:ff:0c.0 > > > > >>[ 1.688490] DMAR: Hardware identity mapping for device 0000:ff:0c.1 > > > > >>[ 1.688491] DMAR: Hardware identity mapping for device 0000:ff:0c.2 > > > > >>[ 1.688493] DMAR: Hardware identity mapping for device 0000:ff:0c.3 > > > > >>[ 1.688494] DMAR: Hardware identity mapping for device 0000:ff:0c.4 > > > > >>[ 1.688496] DMAR: Hardware identity mapping for device 0000:ff:0c.5 > > > > >>[ 1.688498] DMAR: Hardware identity mapping for device 0000:ff:0c.6 > > > > >>[ 1.688499] DMAR: Hardware identity mapping for device 0000:ff:0c.7 > > > > >>[ 1.688500] DMAR: Hardware identity mapping for device 0000:ff:0d.0 > > > > >>[ 1.688502] DMAR: Hardware identity mapping for device 0000:ff:0d.1 > > > > >>[ 1.688504] DMAR: Hardware identity mapping for device 0000:ff:0d.2 > > > > >>[ 1.688505] DMAR: Hardware identity mapping for device 0000:ff:0d.3 > > > > >>[ 1.688506] DMAR: Hardware identity mapping for device 0000:ff:0d.4 > > > > >>[ 1.688508] DMAR: Hardware identity mapping for device 0000:ff:0d.5 > > > > >>[ 1.688509] DMAR: Hardware identity mapping for device 0000:ff:0d.6 > > > > >>[ 1.688510] DMAR: Hardware identity mapping for device 0000:ff:0d.7 > > > > >>[ 1.688511] DMAR: Hardware identity mapping for device 0000:ff:0e.0 > > > > >>[ 1.688513] DMAR: Hardware identity mapping for device 0000:ff:0e.1 > > > > >>[ 1.688514] DMAR: Hardware identity mapping for device 0000:ff:0f.0 > > > > >>[ 1.688516] DMAR: Hardware identity mapping for device 0000:ff:0f.1 > > > > >>[ 1.688517] DMAR: Hardware identity mapping for device 0000:ff:0f.2 > > > > >>[ 1.688518] DMAR: Hardware identity mapping for device 0000:ff:0f.3 > > > > >>[ 1.688520] DMAR: Hardware identity mapping for device 0000:ff:0f.4 > > > > >>[ 1.688521] DMAR: Hardware identity mapping for device 0000:ff:0f.5 > > > > >>[ 1.688523] DMAR: Hardware identity mapping for device 0000:ff:0f.6 > > > > >>[ 1.688524] DMAR: Hardware identity mapping for device 0000:ff:10.0 > > > > >>[ 1.688526] DMAR: Hardware identity mapping for device 0000:ff:10.1 > > > > >>[ 1.688527] DMAR: Hardware identity mapping for device 0000:ff:10.5 > > > > >>[ 1.688529] DMAR: Hardware identity mapping for device 0000:ff:10.6 > > > > >>[ 1.688530] DMAR: Hardware identity mapping for device 0000:ff:10.7 > > > > >>[ 1.688532] DMAR: Hardware identity mapping for device 0000:ff:12.0 > > > > >>[ 1.688533] DMAR: Hardware identity mapping for device 0000:ff:12.1 > > > > >>[ 1.688534] DMAR: Hardware identity mapping for device 0000:ff:12.4 > > > > >>[ 1.688536] DMAR: Hardware identity mapping for device 0000:ff:12.5 > > > > >>[ 1.688537] DMAR: Hardware identity mapping for device 0000:ff:13.0 > > > > >>[ 1.688539] DMAR: Hardware identity mapping for device 0000:ff:13.1 > > > > >>[ 1.688541] DMAR: Hardware identity mapping for device 0000:ff:13.2 > > > > >>[ 1.688542] DMAR: Hardware identity mapping for device 0000:ff:13.3 > > > > >>[ 1.688543] DMAR: Hardware identity mapping for device 0000:ff:13.6 > > > > >>[ 1.688545] DMAR: Hardware identity mapping for device 0000:ff:13.7 > > > > >>[ 1.688546] DMAR: Hardware identity mapping for device 0000:ff:14.0 > > > > >>[ 1.688547] DMAR: Hardware identity mapping for device 0000:ff:14.1 > > > > >>[ 1.688549] DMAR: Hardware identity mapping for device 0000:ff:14.2 > > > > >>[ 1.688550] DMAR: Hardware identity mapping for device 0000:ff:14.3 > > > > >>[ 1.688552] DMAR: Hardware identity mapping for device 0000:ff:14.4 > > > > >>[ 1.688553] DMAR: Hardware identity mapping for device 0000:ff:14.5 > > > > >>[ 1.688559] DMAR: Hardware identity mapping for device 0000:ff:14.6 > > > > >>[ 1.688561] DMAR: Hardware identity mapping for device 0000:ff:14.7 > > > > >>[ 1.688562] DMAR: Hardware identity mapping for device 0000:ff:16.0 > > > > >>[ 1.688564] DMAR: Hardware identity mapping for device 0000:ff:16.1 > > > > >>[ 1.688565] DMAR: Hardware identity mapping for device 0000:ff:16.2 > > > > >>[ 1.688566] DMAR: Hardware identity mapping for device 0000:ff:16.3 > > > > >>[ 1.688568] DMAR: Hardware identity mapping for device 0000:ff:16.6 > > > > >>[ 1.688569] DMAR: Hardware identity mapping for device 0000:ff:16.7 > > > > >>[ 1.688571] DMAR: Hardware identity mapping for device 0000:ff:17.0 > > > > >>[ 1.688573] DMAR: Hardware identity mapping for device 0000:ff:17.1 > > > > >>[ 1.688574] DMAR: Hardware identity mapping for device 0000:ff:17.2 > > > > >>[ 1.688576] DMAR: Hardware identity mapping for device 0000:ff:17.3 > > > > >>[ 1.688577] DMAR: Hardware identity mapping for device 0000:ff:17.4 > > > > >>[ 1.688579] DMAR: Hardware identity mapping for device 0000:ff:17.5 > > > > >>[ 1.688581] DMAR: Hardware identity mapping for device 0000:ff:17.6 > > > > >>[ 1.688582] DMAR: Hardware identity mapping for device 0000:ff:17.7 > > > > >>[ 1.688584] DMAR: Hardware identity mapping for device 0000:ff:1e.0 > > > > >>[ 1.688586] DMAR: Hardware identity mapping for device 0000:ff:1e.1 > > > > >>[ 1.688588] DMAR: Hardware identity mapping for device 0000:ff:1e.2 > > > > >>[ 1.688590] DMAR: Hardware identity mapping for device 0000:ff:1e.3 > > > > >>[ 1.688592] DMAR: Hardware identity mapping for device 0000:ff:1e.4 > > > > >>[ 1.688594] DMAR: Hardware identity mapping for device 0000:ff:1f.0 > > > > >>[ 1.688596] DMAR: Hardware identity mapping for device 0000:ff:1f.2 > > > > >>[ 1.688610] DMAR: Hardware identity mapping for device 0000:7f:08.0 > > > > >>[ 1.688612] DMAR: Hardware identity mapping for device 0000:7f:08.2 > > > > >>[ 1.688614] DMAR: Hardware identity mapping for device 0000:7f:08.3 > > > > >>[ 1.688616] DMAR: Hardware identity mapping for device 0000:7f:09.0 > > > > >>[ 1.688617] DMAR: Hardware identity mapping for device 0000:7f:09.2 > > > > >>[ 1.688619] DMAR: Hardware identity mapping for device 0000:7f:09.3 > > > > >>[ 1.688621] DMAR: Hardware identity mapping for device 0000:7f:0b.0 > > > > >>[ 1.688623] DMAR: Hardware identity mapping for device 0000:7f:0b.1 > > > > >>[ 1.688625] DMAR: Hardware identity mapping for device 0000:7f:0b.2 > > > > >>[ 1.688627] DMAR: Hardware identity mapping for device 0000:7f:0c.0 > > > > >>[ 1.688629] DMAR: Hardware identity mapping for device 0000:7f:0c.1 > > > > >>[ 1.688631] DMAR: Hardware identity mapping for device 0000:7f:0c.2 > > > > >>[ 1.688632] DMAR: Hardware identity mapping for device 0000:7f:0c.3 > > > > >>[ 1.688634] DMAR: Hardware identity mapping for device 0000:7f:0c.4 > > > > >>[ 1.688636] DMAR: Hardware identity mapping for device 0000:7f:0c.5 > > > > >>[ 1.688637] DMAR: Hardware identity mapping for device 0000:7f:0c.6 > > > > >>[ 1.688639] DMAR: Hardware identity mapping for device 0000:7f:0c.7 > > > > >>[ 1.688641] DMAR: Hardware identity mapping for device 0000:7f:0d.0 > > > > >>[ 1.688642] DMAR: Hardware identity mapping for device 0000:7f:0d.1 > > > > >>[ 1.688644] DMAR: Hardware identity mapping for device 0000:7f:0d.2 > > > > >>[ 1.688646] DMAR: Hardware identity mapping for device 0000:7f:0d.3 > > > > >>[ 1.688647] DMAR: Hardware identity mapping for device 0000:7f:0d.4 > > > > >>[ 1.688649] DMAR: Hardware identity mapping for device 0000:7f:0d.5 > > > > >>[ 1.688650] DMAR: Hardware identity mapping for device 0000:7f:0d.6 > > > > >>[ 1.688652] DMAR: Hardware identity mapping for device 0000:7f:0d.7 > > > > >>[ 1.688653] DMAR: Hardware identity mapping for device 0000:7f:0e.0 > > > > >>[ 1.688655] DMAR: Hardware identity mapping for device 0000:7f:0e.1 > > > > >>[ 1.688657] DMAR: Hardware identity mapping for device 0000:7f:0f.0 > > > > >>[ 1.688658] DMAR: Hardware identity mapping for device 0000:7f:0f.1 > > > > >>[ 1.688660] DMAR: Hardware identity mapping for device 0000:7f:0f.2 > > > > >>[ 1.688661] DMAR: Hardware identity mapping for device 0000:7f:0f.3 > > > > >>[ 1.688663] DMAR: Hardware identity mapping for device 0000:7f:0f.4 > > > > >>[ 1.688664] DMAR: Hardware identity mapping for device 0000:7f:0f.5 > > > > >>[ 1.688666] DMAR: Hardware identity mapping for device 0000:7f:0f.6 > > > > >>[ 1.688668] DMAR: Hardware identity mapping for device 0000:7f:10.0 > > > > >>[ 1.688669] DMAR: Hardware identity mapping for device 0000:7f:10.1 > > > > >>[ 1.688671] DMAR: Hardware identity mapping for device 0000:7f:10.5 > > > > >>[ 1.688673] DMAR: Hardware identity mapping for device 0000:7f:10.6 > > > > >>[ 1.688674] DMAR: Hardware identity mapping for device 0000:7f:10.7 > > > > >>[ 1.688676] DMAR: Hardware identity mapping for device 0000:7f:12.0 > > > > >>[ 1.688678] DMAR: Hardware identity mapping for device 0000:7f:12.1 > > > > >>[ 1.688684] DMAR: Hardware identity mapping for device 0000:7f:12.4 > > > > >>[ 1.688686] DMAR: Hardware identity mapping for device 0000:7f:12.5 > > > > >>[ 1.688688] DMAR: Hardware identity mapping for device 0000:7f:13.0 > > > > >>[ 1.688689] DMAR: Hardware identity mapping for device 0000:7f:13.1 > > > > >>[ 1.688691] DMAR: Hardware identity mapping for device 0000:7f:13.2 > > > > >>[ 1.688692] DMAR: Hardware identity mapping for device 0000:7f:13.3 > > > > >>[ 1.688694] DMAR: Hardware identity mapping for device 0000:7f:13.6 > > > > >>[ 1.688696] DMAR: Hardware identity mapping for device 0000:7f:13.7 > > > > >>[ 1.688697] DMAR: Hardware identity mapping for device 0000:7f:14.0 > > > > >>[ 1.688699] DMAR: Hardware identity mapping for device 0000:7f:14.1 > > > > >>[ 1.688701] DMAR: Hardware identity mapping for device 0000:7f:14.2 > > > > >>[ 1.688703] DMAR: Hardware identity mapping for device 0000:7f:14.3 > > > > >>[ 1.688704] DMAR: Hardware identity mapping for device 0000:7f:14.4 > > > > >>[ 1.688706] DMAR: Hardware identity mapping for device 0000:7f:14.5 > > > > >>[ 1.688708] DMAR: Hardware identity mapping for device 0000:7f:14.6 > > > > >>[ 1.688710] DMAR: Hardware identity mapping for device 0000:7f:14.7 > > > > >>[ 1.688712] DMAR: Hardware identity mapping for device 0000:7f:16.0 > > > > >>[ 1.688714] DMAR: Hardware identity mapping for device 0000:7f:16.1 > > > > >>[ 1.688716] DMAR: Hardware identity mapping for device 0000:7f:16.2 > > > > >>[ 1.688718] DMAR: Hardware identity mapping for device 0000:7f:16.3 > > > > >>[ 1.688720] DMAR: Hardware identity mapping for device 0000:7f:16.6 > > > > >>[ 1.688722] DMAR: Hardware identity mapping for device 0000:7f:16.7 > > > > >>[ 1.688724] DMAR: Hardware identity mapping for device 0000:7f:17.0 > > > > >>[ 1.688726] DMAR: Hardware identity mapping for device 0000:7f:17.1 > > > > >>[ 1.688729] DMAR: Hardware identity mapping for device 0000:7f:17.2 > > > > >>[ 1.688731] DMAR: Hardware identity mapping for device 0000:7f:17.3 > > > > >>[ 1.688733] DMAR: Hardware identity mapping for device 0000:7f:17.4 > > > > >>[ 1.688735] DMAR: Hardware identity mapping for device 0000:7f:17.5 > > > > >>[ 1.688737] DMAR: Hardware identity mapping for device 0000:7f:17.6 > > > > >>[ 1.688739] DMAR: Hardware identity mapping for device 0000:7f:17.7 > > > > >>[ 1.688741] DMAR: Hardware identity mapping for device 0000:7f:1e.0 > > > > >>[ 1.688742] DMAR: Hardware identity mapping for device 0000:7f:1e.1 > > > > >>[ 1.688744] DMAR: Hardware identity mapping for device 0000:7f:1e.2 > > > > >>[ 1.688746] DMAR: Hardware identity mapping for device 0000:7f:1e.3 > > > > >>[ 1.688748] DMAR: Hardware identity mapping for device 0000:7f:1e.4 > > > > >>[ 1.688750] DMAR: Hardware identity mapping for device 0000:7f:1f.0 > > > > >>[ 1.688752] DMAR: Hardware identity mapping for device 0000:7f:1f.2 > > > > >>[ 1.688769] DMAR: Hardware identity mapping for device > 0000:00:00.0 > > > > >>[ 1.688774] DMAR: Hardware identity mapping for device > 0000:00:01.0 > > > > >>[ 1.688778] DMAR: Hardware identity mapping for device > 0000:00:01.1 > > > > >>[ 1.688783] DMAR: Hardware identity mapping for device > 0000:00:03.0 > > > > >>[ 1.688785] DMAR: Hardware identity mapping for device > 0000:00:04.0 > > > > >>[ 1.688787] DMAR: Hardware identity mapping for device > 0000:00:04.1 > > > > >>[ 1.688790] DMAR: Hardware identity mapping for device > 0000:00:04.2 > > > > >>[ 1.688792] DMAR: Hardware identity mapping for device > 0000:00:04.3 > > > > >>[ 1.688794] DMAR: Hardware identity mapping for device > 0000:00:04.4 > > > > >>[ 1.688797] DMAR: Hardware identity mapping for device > 0000:00:04.5 > > > > >>[ 1.688799] DMAR: Hardware identity mapping for device > 0000:00:04.6 > > > > >>[ 1.688801] DMAR: Hardware identity mapping for device > 0000:00:04.7 > > > > >>[ 1.688803] DMAR: Hardware identity mapping for device > 0000:00:05.0 > > > > >>[ 1.688807] DMAR: Hardware identity mapping for device > 0000:00:05.1 > > > > >>[ 1.688809] DMAR: Hardware identity mapping for device > 0000:00:05.2 > > > > >>[ 1.688811] DMAR: Hardware identity mapping for device > 0000:00:05.4 > > > > >>[ 1.688814] DMAR: Hardware identity mapping for device > 0000:00:11.0 > > > > >>[ 1.688816] DMAR: Hardware identity mapping for device > 0000:00:11.4 > > > > >>[ 1.688818] DMAR: Hardware identity mapping for device > 0000:00:14.0 > > > > >>[ 1.688821] DMAR: Hardware identity mapping for device > 0000:00:16.0 > > > > >>[ 1.688823] DMAR: Hardware identity mapping for device > 0000:00:16.1 > > > > >>[ 1.688825] DMAR: Hardware identity mapping for device > 0000:00:1a.0 > > > > >>[ 1.688828] DMAR: Hardware identity mapping for device 0000:00:1c.0 > > > > >>[ 1.688831] DMAR: Hardware identity mapping for device 0000:00:1c.2 > > > > >>[ 1.688833] DMAR: Hardware identity mapping for device > 0000:00:1d.0 > > > > >>[ 1.688835] DMAR: Hardware identity mapping for device 0000:00:1f.0 > > > > >>[ 1.688837] DMAR: Hardware identity mapping for device 0000:00:1f.2 > > > > >>[ 1.688844] DMAR: Hardware identity mapping for device 0000:00:1f.3 > > > > >>[ 1.688846] DMAR: Hardware identity mapping for device 0000:00:1f.6 > > > > >>[ 1.688864] DMAR: Hardware identity mapping for device > 0000:01:00.0 > > > > >>[ 1.688870] DMAR: Hardware identity mapping for device > 0000:01:00.1 > > > > >>[ 1.688876] DMAR: Hardware identity mapping for device > 0000:01:00.2 > > > > >>[ 1.688883] DMAR: Hardware identity mapping for device > 0000:01:00.3 > > > > >>[ 1.688895] DMAR: Hardware identity mapping for device > 0000:80:01.0 > > > > >>[ 1.688899] DMAR: Hardware identity mapping for device > 0000:80:02.0 > > > > >>[ 1.688903] DMAR: Hardware identity mapping for device > 0000:80:03.0 > > > > >>[ 1.688906] DMAR: Hardware identity mapping for device > 0000:80:04.0 > > > > >>[ 1.688908] DMAR: Hardware identity mapping for device > 0000:80:04.1 > > > > >>[ 1.688911] DMAR: Hardware identity mapping for device > 0000:80:04.2 > > > > >>[ 1.688913] DMAR: Hardware identity mapping for device > 0000:80:04.3 > > > > >>[ 1.688915] DMAR: Hardware identity mapping for device > 0000:80:04.4 > > > > >>[ 1.688918] DMAR: Hardware identity mapping for device > 0000:80:04.5 > > > > >>[ 1.688920] DMAR: Hardware identity mapping for device > 0000:80:04.6 > > > > >>[ 1.688923] DMAR: Hardware identity mapping for device > 0000:80:04.7 > > > > >>[ 1.688936] DMAR: Hardware identity mapping for device > 0000:80:05.0 > > > > >>[ 1.688940] DMAR: Hardware identity mapping for device > 0000:80:05.1 > > > > >>[ 1.688943] DMAR: Hardware identity mapping for device > 0000:80:05.2 > > > > >>[ 1.688945] DMAR: Hardware identity mapping for device > 0000:80:05.4 > > > > >>[ 1.688956] DMAR: Hardware identity mapping for device > 0000:81:00.0 > > > > >>[ 1.688961] DMAR: Hardware identity mapping for device > 0000:81:00.1 > > > > >>[ 1.688974] DMAR: Hardware identity mapping for device > 0000:82:00.0 > > > > >>[ 1.688980] DMAR: Hardware identity mapping for device > 0000:82:00.1 > > > > >>[ 1.688986] DMAR: Hardware identity mapping for device > 0000:82:00.2 > > > > >>[ 1.688992] DMAR: Hardware identity mapping for device > 0000:82:00.3 > > > > >>[ 1.689006] DMAR: Hardware identity mapping for device > 0000:83:00.0 > > > > >>[ 1.689012] DMAR: Hardware identity mapping for device > 0000:83:00.1 > > > > >>[ 1.689018] DMAR: Hardware identity mapping for device > 0000:83:00.2 > > > > >>[ 1.689024] DMAR: Hardware identity mapping for device > 0000:83:00.3 > > > > >>[ 1.689025] DMAR: Setting RMRR: > > > > >>[ 1.689026] DMAR: Ignoring identity map for HW passthrough device > > > > >>0000:00:14.0 [0x7ba75000 - 0x7ba84fff] > > > > >>[ 1.689027] DMAR: Ignoring identity map for HW passthrough device > > > > >>0000:00:1a.0 [0x7ba75000 - 0x7ba84fff] > > > > >>[ 1.689028] DMAR: Ignoring identity map for HW passthrough device > > > > >>0000:00:1d.0 [0x7ba75000 - 0x7ba84fff] > > > > >>[ 1.689036] DMAR: Prepare 0-16MiB unity mapping for LPC > > > > >>[ 1.689036] DMAR: Ignoring identity map for HW passthrough device > > > > >>0000:00:1f.0 [0x0 - 0xffffff] > > > > >>[ 1.689040] DMAR: Intel(R) Virtualization Technology for Directed > > I/O > > > > >>[ 2.299766] DMAR: 32bit 0000:00:11.4 uses non-identity mapping > > > > >>[ 2.301941] DMAR: 32bit 0000:00:1a.0 uses non-identity mapping > > > > >>[ 2.302679] DMAR: Setting identity map for device 0000:00:1a.0 > > [0x7ba75000 > > > > >- > > > > >>0x7ba84fff] > > > > >>[ 2.320910] DMAR: 32bit 0000:00:1d.0 uses non-identity mapping > > > > >>[ 2.321685] DMAR: Setting identity map for device 0000:00:1d.0 > > > > >[0x7ba75000 > > > > >>- 0x7ba84fff] > > > > >># > > > > >> > > > > >> > > > > >>On Tue, Mar 28, 2017 at 1:46 AM, Bodireddy, Bhanuprakash > > > > >><bhanuprakash.bodire...@intel.com> wrote: > > > > >>Hello shivaram, > > > > >> > > > > >>Darrell has already asked if VT-d is enabled? > > > > >>Your cmdline options are appropriate as you have iommu > > > > >>enabled(intel_iommu=on iommu=pt). > > > > >>can you also check the output of below mentioned command to see > > if VT-d > > > > >is > > > > >>enabled in BIOS? > > > > >> > > > > >>$ dmesg | grep -e DMAR -e IOMMU > > > > >> > > > > >>Regards, > > > > >>Bhanuprakash. > > > > >> > > > > >>From: ovs-discuss-boun...@openvswitch.org [mailto:ovs-discuss- > > > > >>boun...@openvswitch.org] On Behalf Of Shivaram Mysore > > > > >>Sent: Tuesday, March 28, 2017 6:25 AM > > > > >>To: Darrell Ball <db...@vmware.com> > > > > >>Cc: ovs-discuss@openvswitch.org > > > > >>Subject: Re: [ovs-discuss] Error attaching device to DPDK > > > > >> > > > > >># uname -r > > > > >>4.10.0-14-generic > > > > >> > > > > >># cat /proc/cmdline > > > > >>BOOT_IMAGE=/boot/vmlinuz-4.10.0-14-generic > > > > >root=/dev/mapper/intelsdn- > > > > >>-vg-root ro quiet intel_iommu=on iommu=pt default_hugepagesz=1G > > > > >>hugepagesz=1G hugepages=4 > > > > >> > > > > >># uname -a > > > > >>Linux intelsdn 4.10.0-14-generic #16-Ubuntu SMP Fri Mar 17 > > 15:19:26 UTC > > > > >2017 > > > > >>x86_64 x86_64 x86_64 GNU/Linux > > > > >> > > > > >># cat /etc/*release > > > > >>DISTRIB_ID=Ubuntu > > > > >>DISTRIB_RELEASE=17.04 > > > > >>DISTRIB_CODENAME=zesty > > > > >>DISTRIB_DESCRIPTION="Ubuntu Zesty Zapus (development branch)" > > > > >>NAME="Ubuntu" > > > > >>VERSION="17.04 (Zesty Zapus)" > > > > >>ID=ubuntu > > > > >>ID_LIKE=debian > > > > >>PRETTY_NAME="Ubuntu Zesty Zapus (development branch)" > > > > >>VERSION_ID="17.04" > > > > >> > > > > >> > > > > >>Nothing in the syslog related > > > > >> > > > > >>On Mon, Mar 27, 2017 at 10:02 PM, Darrell Ball > > <db...@vmware.com> > > wrote: > > > > >>I see the pastebins now. > > > > >> > > > > >>What kernel version are you using ? > > > > >>VT-d enabled ? > > > > >>Anything related in syslog ? > > > > >> > > > > >>From: Shivaram Mysore <shivaram.mys...@gmail.com> > > > > >>Date: Monday, March 27, 2017 at 9:34 PM > > > > >>To: Darrell Ball <db...@vmware.com> > > > > >>Cc: "ovs-discuss@openvswitch.org" <ovs-discuss@openvswitch.org> > > > > >>Subject: Re: [ovs-discuss] Error attaching device to DPDK > > > > >> > > > > >>One more datapoint, please check the paste bin log in URL sent > > > > >earlier. Dpdk- > > > > >>devbind works and you can see results in the paste bin log > > > > >> > > > > >>/Shivaram > > > > >>::Sent from my mobile device:: > > > > >> > > > > >>On Mar 27, 2017, at 8:34 PM, Darrell Ball <db...@vmware.com> wrote: > > > > >>see inline below > > > > >> > > > > >>But a few questions: > > > > >>1) Are you using the recommended version of dpdk with OVS 2.7 – > > 16.11 ? > > > > >>2) Are you getting a crash still or is this post crash ? > > > > >>3) Was the previous crash still happening with 16.11 or only > > 17.02 ? > > > > >> > > > > >>Thanks > > > > >>Darrell > > > > >> > > > > >>From: <ovs-discuss-boun...@openvswitch.org> on behalf of > > Shivaram > > > > >>Mysore <shivaram.mys...@gmail.com> > > > > >>Date: Monday, March 27, 2017 at 7:45 PM > > > > >>To: "ovs-discuss@openvswitch.org" <ovs-discuss@openvswitch.org> > > > > >>Subject: [ovs-discuss] Error attaching device to DPDK > > > > >> > > > > >>Hello, > > > > >> > > > > >>I have installed OVS 2.7 running. When I do: > > > > >>1. # ovs-vsctl add-port ovs-ip64-br0 dpdk-p0 -- set Interface > > dpdk-p0 > > > > >>type=dpdk options:dpdk-devargs=0000:82:00.0 > > > > >>2. ovs-vsctl: Error detected while setting up 'dpdk-p0': Error > attaching > > > > >>device '0000:82:00.0' to DPDK. See ovs-vswitchd log for details. > > > > >> > > > > >>Possibly, the port is not bound. > > > > >>Check using > > > > >>$DPDK_DIR/tools/dpdk-devbind.py –status > > > > >>and see the documentation for more detail > > > > >> > > > > >>and the log file shows: > > > > >>1. 2017-03-28T02:04:40.004Z|00064|netdev_dpdk|WARN|Error > > attaching > > > > >>device '0000:82:00.0' to DPDK > > > > >>2. 2017-03-28T02:04:40.004Z|00065|netdev|WARN|dpdk-p0: could > > > > >>not set configuration (Invalid argument) > > > > >> > > > > >>What would cause such a situation? > > > > >> > > > > >>Commands run: https://urldefense.proofpoint.com/v2/url?u=https- > > > 3A__pastebin.com_0P754x0t&d=DwIGaQ&c=uilaK90D4TOVoH58JNXRgQ&r= > > BVhFA09CGX7JQ5Ih- > > > uZnsw&m=IKHiqOFj5oogxEfx6OMgK9IHxp2aktbssMBTZUoA2P0&s=XEhP0isS > > M6FAfGw-HXbOfMLfomYnhSo1G2UxvC1qjp8&e= > > > > >>Detailed OVS Log: > > https://urldefense.proofpoint.com/v2/url?u=https- > > > 3A__pastebin.com_ctBhLuRG&d=DwIGaQ&c=uilaK90D4TOVoH58JNXRgQ&r= > > BVhFA09CGX7JQ5Ih- > > > uZnsw&m=IKHiqOFj5oogxEfx6OMgK9IHxp2aktbssMBTZUoA2P0&s=90tudSR > > NE_HlxrCclX83wTmqlUL5xMPrRha7_-fMAKs&e= > > > > >> > > > > >> > > > > > > > > > > > > > > > > > _______________________________________________ > > > > discuss mailing list > > > > disc...@openvswitch.org > > > > https://urldefense.proofpoint.com/v2/url?u=https- > > 3A__mail.openvswitch.org_mailman_listinfo_ovs- > > > 2Ddiscuss&d=DwIGaQ&c=uilaK90D4TOVoH58JNXRgQ&r=BVhFA09CGX7JQ5Ih > > -uZnsw&m=IKHiqOFj5oogxEfx6OMgK9IHxp2aktbssMBTZUoA2P0&s=9- > > oI5HFUKs-b_kDAFcnn2vFwTcuE2xw2SFvd-zE6vWY&e= > > > > > > > > > > > > _______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
