[dpdk-users] Cannot run Quick Start on ubuntu16.04.1 VM

[Marco Kwok]

Hello all,

I am running vmware workstation for ubuntu 16.04.1.
Following the guide here:
http://dpdk.org/doc/quick-start

everything is fine until this line:

> echo 64 > /sys/devices/system/node/node0/hugepages/hugepages-
> 2048kB/nr_hugepages

I got permission denied even with sudo, so I get through it with:

> sh -c "echo 64 > /sys/devices/system/node/node0/hugepages/hugepages-
> 2048kB/nr_hugepages"


Then I want to start the program testpmd and I found the argument of this
line is not supported:

> -i --nb-cores=2 --nb-ports=2 --total-num-mbufs=2048


With this command to start the testpmd(my card is not eth0 and eth1 but
ens33 and 38 instead):
sudo build/app/testpmd -c7 -n3 --vdev=eth_pcap0,iface=ens33
--vdev=eth_pcap1,iface=ens38

EAL: Detected 4 lcore(s)
>
> EAL: No free hugepages reported in hugepages-1048576kB
>
> EAL: Probing VFIO support...
>
> PMD: Initializing pmd_pcap for eth_pcap0
>
> PMD: Creating pcap-backed ethdev on numa socket 0
>
> PMD: Initializing pmd_pcap for eth_pcap1
>
> PMD: Creating pcap-backed ethdev on numa socket 0
>
> PMD: bnxt_rte_pmd_init() called for (null)
>
> EAL: PCI device :02:01.0 on NUMA socket -1
>
> EAL:   probe driver: 8086:100f rte_em_pmd
>
> EAL: PCI device :02:06.0 on NUMA socket -1
>
> EAL:   probe driver: 8086:100f rte_em_pmd
>
> USER1: create a new mbuf pool : n=163456, size=2176,
>> socket=0
>
> EAL: Error - exiting with code: 1
>
>   Cause: Creation of mbuf pool for socket 0 failed: Cannot allocate memory
>
>
Does anybody know how I could get through this?

Best Regards,
Mark

Re: [dpdk-users] Mellanox rss hash information

[Saleh, Omran]

I am still getting the same problem with the optimized function, the result 
value from  rte_softrss_be is different than mbuf.hash->rss.
 any idea about the problem, can be an old kernel produce this problem?

-Original Message-
From: Shahaf Shuler [mailto:shah...@mellanox.com] 
Sent: Mittwoch, 28. Dezember 2016 12:00
To: Saleh, Omran ; users@dpdk.org
Subject: RE: Mellanox rss hash information

Have you tried you use rte_softrss_be function?

I suspect you have an endianness issue here as the mlx5 rss_hash_default_key is 
already in big-endian format. 


/** 
 * Optimized implementation.
 * If you want the calculated hash value matches NIC RSS value
 * you have to use special converted key with rte_convert_rss_key() fn.
 * @param input_tuple
 *   Pointer to input tuple
 * @param input_len
 *   Length of input_tuple in 4-bytes chunks
 * @param *rss_key
 *   Pointer to RSS hash key.
 * @return
 *   Calculated hash value.
 */
static inline uint32_t
rte_softrss_be(uint32_t *input_tuple, uint32_t input_len,
const uint8_t *rss_key)
{
...
}
--Shahaf

-Original Message-
From: users [mailto:users-boun...@dpdk.org] On Behalf Of Saleh, Omran
Sent: Wednesday, December 28, 2016 10:29 AM
To: users@dpdk.org
Subject: [dpdk-users] Mellanox rss hash information

Hi All,
Currently, I am using DPDK 16.11 with Mellanox NIC adapters (mlx5). I encounter 
a problem that the RSS hash value from mbuf  (mbug.hash->rss) is not the same 
as calculated from rte_softrss(). It is completely different.
I am using the default hash key from mlx5

uint8_t rss_hash_default_key[] = {
0x2c, 0xc6, 0x81, 0xd1,
0x5b, 0xdb, 0xf4, 0xf7,
0xfc, 0xa2, 0x83, 0x19,
0xdb, 0x1a, 0x3e, 0x94,
0x6b, 0x9e, 0x38, 0xd9,
0x2c, 0x9c, 0x03, 0xd1,
0xad, 0x99, 0x44, 0xa7,
0xd9, 0x56, 0x3d, 0x59,
0x06, 0x3c, 0x25, 0xf3,
0xfc, 0x1f, 0xdc, 0x2a,
};
And the following call:
auto hash = rte_softrss(_addr, RTE_THASH_V4_L4_LEN, 
rss_hash_default_key); with RSS configurations as the following:

port_conf.rxmode.mq_mode = ETH_MQ_RX_RSS; port_conf.rx_adv_conf.rss_conf.rss_hf 
= ETH_RSS_NONFRAG_IPV4_TCP; port_conf.rx_adv_conf.rss_conf.rss_key= 
rss_hash_default_key; port_conf.rx_adv_conf.rss_conf.rss_key_len = 40;

This is working perfectly with ixgbe  but Mellanox not. Can you help me to 
figure out what is the problem? What is the default Toeplitz function for mlx5?!

My NIC adapters are  Mellanox Technologies MT27700 Family [ConnectX-4] With 
firmware version 12.16.1006 and ofed version  3.3-2.0.0

uname -a : Linux ibmhanar16 3.12.49-11-default #1 SMP Wed Nov 11 20:52:43 UTC 
2015 (8d714a0) x86_64 x86_64 x86_64 GNU/Linux

Thank you
Best regards,
Omran Saleh

Re: [dpdk-users] DPDK: Inter VM communication of iperf3 TCP throughput is very low on same host compare to non DPDK throughput

[Bodireddy, Bhanuprakash]

>-Original Message-
>From: Rajalakshmi Prabhakar [mailto:krajalaks...@tataelxsi.co.in]
>Sent: Tuesday, December 27, 2016 9:52 AM
>To: d...@dpdk.org; users@dpdk.org
>Cc: Bodireddy, Bhanuprakash 
>Subject: DPDK: Inter VM communication of iperf3 TCP throughput is very low
>on same host compare to non DPDK throughput
>
>Hello,
>Kindly Support me to get high throughput in inter VM communication of iperf3
>TCP in OpenStack DPDK host. I am not sure that I am mailing to the right ID
>sorry for the inconvenience.
OVS mailing list should be the appropriate one for the problem you reported 
here. 
Use  ovs-disc...@openvswitch.org (or) d...@openvswitch.org. 

>
>Host - ubuntu16.04
>devstack - stable/newton
>which install DPDK 16.07 and OVS 2.6 versions
>with DPDK plugin and following DPDK configurations
>Grub changes
>GRUB_CMDLINE_LINUX_DEFAULT="quiet splash default_hugepagesz=1G
>hugepagesz=1G hugepages=8 iommu=pt intel_iommu=on"
>local.conf - changes for DPDK
>enable_plugin networking-ovs-dpdk
>https://git.openstack.org/openstack/networking-ovs-dpdk master
>OVS_DPDK_MODE=controller_ovs_dpdk
>OVS_NUM_HUGEPAGES=8
>OVS_CORE_MASK=2
>OVS_PMD_CORE_MASK=4
Only one PMD core is used in your case, scaling the PMD threads can be one 
option for higher throughputs. 

>OVS_DPDK_BIND_PORT=False
>OVS_SOCKET_MEM=2048
>OVS_DPDK_VHOST_USER_DEBUG=n
>OVS_ALLOCATE_HUGEPAGES=True
>OVS_HUGEPAGE_MOUNT_PAGESIZE=1G
>MULTI_HOST=1
>OVS_DATAPATH_TYPE=netdev
>before VM creation
>#nova flavor-key m1.small set hw:mem_page_size=1048576
>Able to create two ubuntu instance in flavor m1.small
How many cores are assigned for the VM and have you tried CPU pinning options 
instead of allowing the threads to float across the cores?

>Achieved iperf3 tcp throughput of ~7.5Gbps
Are you seeing high drops at the vHost ports and retransmissions?  Do you see 
the same throughput difference with UDP traffic?
However I can't explain now the throughput gap you are observing here. Couple 
of things worth checking

- For thread starvation (htop to see thread activity on the cores)
-  I see that you have single socket setup and no QPI involved. As you have HT 
enabled, check if appropriate thread siblings are used.
- Check pmd thread/port statistics for anomaly. 

BTW, the responses can be slow at this point due to yearend vacation. 

Regards,
Bhanuprakash. 

>Ensured the vhostport is created and HugePage is consumed at the end of
>2VM created each of 2GB ie 4GB for VMs and 2GB for socket totally 6GB
>$ sudo cat /proc/meminfo |grep Huge
>AnonHugePages: 0 kB
>HugePages_Total: 8
>HugePages_Free: 2
>HugePages_Rsvd: 0
>HugePages_Surp: 0
>Hugepagesize: 1048576 kB
>The same scenario carried for without DPDK case of openstack and achieved
>higher throughput of ~19Gbps, which is contradictory to the expected results.
>Kindly suggest me what additional DPDK configuration to be done for high
>throughput. Also tried cpu pinning and multi queue for OpenStack DPDK but
>no improvement in the result.
>Test PC is single NUMA only.I am not doing NIC binding as only trying to
>validate inter-VM communication in same host. PFB my PC configurations.
>$ lscpu
>Architecture: x86_64
>CPU op-mode(s): 32-bit, 64-bit
>Byte Order: Little Endian
>CPU(s): 12
>On-line CPU(s) list: 0-11
>Thread(s) per core: 2
>Core(s) per socket: 6
>Socket(s): 1
>NUMA node(s): 1
>Vendor ID: GenuineIntel
>CPU family: 6
>Model: 63
>Model name: Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
>Stepping: 2
>CPU MHz: 1212.000
>CPU max MHz: 2400.
>CPU min MHz: 1200.
>BogoMIPS: 4794.08
>Virtualization: VT-x
>L1d cache: 32K
>L1i cache: 32K
>L2 cache: 256K
>L3 cache: 15360K
>NUMA node0 CPU(s): 0-11
>Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat
>pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1g b
>rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology
>nonstop_t sc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx
>smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic
>movbe popcnt tsc_deadline _timer aes xsave avx f16c rdrand lahf_lm abm
>epb tpr_shadow vnmi flexpriority ep t vpid fsgsbase tsc_adjust bmi1 avx2
>smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat
>pln pts
>I am following INSTALL.DPDK.ADVANCED.md but no clue on low throughput.
>
>
>Best Regards
>Rajalakshmi Prabhakar
>
>Specialist - Communication BU | Wireless Division
>TATA ELXSI
>IITM Research park , Kanagam road ,  Taramani ,   Chennai 600 113   India
>Tel +91 44 66775031   Cell +91 9789832957
>www.tataelxsi.com

[dpdk-users] running pkt-gen in a container

[Avi Cohen]

Hello
I have the following setup:
Ovs-dpdk is running in a VM.
I have 1 dpdk type port in the OVS (connected to the NIC w SR-IOV). 
 I  have an LXC-container (or for simplicity a network namespace) in the VM - 
currently connected to the OVS via veth-pair. 
 I want to run pkt-get in the namespace -  my questions:

1. I guess I have to run another DPDK instance in the namespace - to run 
pkt-gen on top of DPD, correct ?  (i.e. - cannot use the ovs-dpdk instance ) 2. 
How to connect the namespace to the OVS in order to achieve zero copy ? (I 
guess veth is not suitable for that)  - can I use vHost-user netdev or ivshmem, 
 to connet ovs with the namespace ?
3. In cont. to question 2 - what interface should I bind to the DPDK instance 
running in the namespace Thanks in advance Avi 


 

Re: [dpdk-users] DPDK pinned Memory for non Root User

[Anupam Kapoor]

On Wed, Dec 28, 2016 at 4:21 PM, Shreyansh Jain 
wrote:

> There is possible way, as documented here:
> http://dpdk.org/doc/guides/linux_gsg/enable_func.html#
> running-dpdk-applications-without-root-privileges
>
> But I haven't tried it myself and neither do I know if it will solve your
> problem.
>

​yeah it says that but afaik, ​/proc/self/pagemap cannot be opened (during
init) due to virt2phys translation issues...

​---
thanks
anupam​


In the beginning was the lambda, and the lambda was with Emacs, and Emacs
was the lambda.

Re: [dpdk-users] Mellanox rss hash information

[Shahaf Shuler]

Have you tried you use rte_softrss_be function?

I suspect you have an endianness issue here as the mlx5 rss_hash_default_key is 
already in big-endian format. 


/** 
 * Optimized implementation.
 * If you want the calculated hash value matches NIC RSS value
 * you have to use special converted key with rte_convert_rss_key() fn.
 * @param input_tuple
 *   Pointer to input tuple
 * @param input_len
 *   Length of input_tuple in 4-bytes chunks
 * @param *rss_key
 *   Pointer to RSS hash key.
 * @return
 *   Calculated hash value.
 */
static inline uint32_t
rte_softrss_be(uint32_t *input_tuple, uint32_t input_len,
const uint8_t *rss_key)
{
...
}
--Shahaf

-Original Message-
From: users [mailto:users-boun...@dpdk.org] On Behalf Of Saleh, Omran
Sent: Wednesday, December 28, 2016 10:29 AM
To: users@dpdk.org
Subject: [dpdk-users] Mellanox rss hash information

Hi All,
Currently, I am using DPDK 16.11 with Mellanox NIC adapters (mlx5). I encounter 
a problem that the RSS hash value from mbuf  (mbug.hash->rss) is not the same 
as calculated from rte_softrss(). It is completely different.
I am using the default hash key from mlx5

uint8_t rss_hash_default_key[] = {
0x2c, 0xc6, 0x81, 0xd1,
0x5b, 0xdb, 0xf4, 0xf7,
0xfc, 0xa2, 0x83, 0x19,
0xdb, 0x1a, 0x3e, 0x94,
0x6b, 0x9e, 0x38, 0xd9,
0x2c, 0x9c, 0x03, 0xd1,
0xad, 0x99, 0x44, 0xa7,
0xd9, 0x56, 0x3d, 0x59,
0x06, 0x3c, 0x25, 0xf3,
0xfc, 0x1f, 0xdc, 0x2a,
};
And the following call:
auto hash = rte_softrss(_addr, RTE_THASH_V4_L4_LEN, 
rss_hash_default_key); with RSS configurations as the following:

port_conf.rxmode.mq_mode = ETH_MQ_RX_RSS; port_conf.rx_adv_conf.rss_conf.rss_hf 
= ETH_RSS_NONFRAG_IPV4_TCP; port_conf.rx_adv_conf.rss_conf.rss_key= 
rss_hash_default_key; port_conf.rx_adv_conf.rss_conf.rss_key_len = 40;

This is working perfectly with ixgbe  but Mellanox not. Can you help me to 
figure out what is the problem? What is the default Toeplitz function for mlx5?!

My NIC adapters are  Mellanox Technologies MT27700 Family [ConnectX-4] With 
firmware version 12.16.1006 and ofed version  3.3-2.0.0

uname -a : Linux ibmhanar16 3.12.49-11-default #1 SMP Wed Nov 11 20:52:43 UTC 
2015 (8d714a0) x86_64 x86_64 x86_64 GNU/Linux

Thank you
Best regards,
Omran Saleh

Re: [dpdk-users] DPDK pinned Memory for non Root User

[Shreyansh Jain]

Karthi,

> -Original Message-
> From: users [mailto:users-boun...@dpdk.org] On Behalf Of Anupam Kapoor
> Sent: Wednesday, December 28, 2016 4:15 PM
> To: Karthi M 
> Cc: users@dpdk.org
> Subject: Re: [dpdk-users] DPDK pinned Memory for non Root User
> 
> On Wed, Dec 28, 2016 at 3:51 PM, Karthi M  wrote:
> 
> > So is there a way for App to get pinned memory for the users other than
> > the root user.
> 
> 
> i am assuming that you are on linux. ​afaik there is no way to ​run dpdk as
> non-root. the thread you linked to has necessary information...

There is possible way, as documented here:
http://dpdk.org/doc/guides/linux_gsg/enable_func.html#running-dpdk-applications-without-root-privileges

But I haven't tried it myself and neither do I know if it will solve your 
problem.

> 
> --
> thanks
> anupam
> 
> 
> In the beginning was the lambda, and the lambda was with Emacs, and Emacs
> was the lambda.

Re: [dpdk-users] DPDK pinned Memory for non Root User

[Anupam Kapoor]

On Wed, Dec 28, 2016 at 3:51 PM, Karthi M  wrote:

> So is there a way for App to get pinned memory for the users other than
> the root user.


i am assuming that you are on linux. ​afaik there is no way to ​run dpdk as
non-root. the thread you linked to has necessary information...

--
thanks
anupam


In the beginning was the lambda, and the lambda was with Emacs, and Emacs
was the lambda.

[dpdk-users] DPDK pinned Memory for non Root User

[Karthi M]

Hi All, 

I’m trying to run DPDK examples/App in a non-root user mode. 
rte_mempool_init always returns 0MB Available memory. But if I run the same as 
a root user, everything works fine. So I’m not able to get the pinned memory 
for the App to start as a unprivileged user. So is there a way for App to get 
pinned memory for the users other than the root user.  Can Someone help me out 
on this and I have come across some similar issue DPDK mail thread for the same 
issue.

http://dpdk.org/ml/archives/users/2016-July/000709.html 





Regards, 

Karthi | +91 9036339210
CloudSimple Inc 

Re: [dpdk-users] Dpdk poor performance on virtual machine

[edgar helmut]

I tried this procedure as well as few others.
at all of the procedures the HugePages_Free or HugePages_Rsvd doesn't
change when creating the guest, though only AnonHugePages increases.
I am struggling with it for a while without real success.
I do suspect that this anonymous hugepages doesn't really make the job.

any idea?


On Tue, Dec 27, 2016 at 8:52 PM Stephen Hemminger <
step...@networkplumber.org> wrote:

> On Tue, 27 Dec 2016 15:59:08 +
> edgar helmut  wrote:
>
> > short explanation for how to read the comparison:
> > first row is packet length
> > throughput is half duplex, means:
> > second row is vm throughput of port 1 to 2 (port 2 to 1 has approximately
> > same throughput) in gbps.
> > third row is host throughput of port 1 to 2 (port 2 to 1 has
> approximately
> > same throughput) in gbps.
> >
> > i.e. on 1500 bytes packet size testpmd delivers ~9.82 gbps from port 1
> to 2
> > and another ~9.82 gbps from port 2 to 1, while at the vm it only delivers
> > ~3.9 gbps for each direction.
> >
> >
> > On Tue, Dec 27, 2016 at 5:52 PM edgar helmut 
> > wrote:
> >
> > > Thanks. That's the document i am following.
> > > For the best i can only ask that the hugepages won't be shared with
> > > others, but it never reserve it from the pre allocated hugepages of the
> > > host.
> > > Did you have a chance to use hugepages for a guest
> > >
> > > as for the interfaces, i am using the virtio/vhost which creates the
> > > macvtap:
> > > 
> > > 
> > > 
> > > 
> > > 
> > >  > > function='0x0'/>
> > > 
> > >
> > > The following is a performance comparison host vs. vm using testpmd. as
> > > you can see vm performance is poor.
> > >
> > > (sudo x86_64-native-linuxapp-gcc/app/testpmd -c 0x1f -n 3 -m 1024 --
> > > --coremask=0x1e --portmask=3 -i)
> > >
> > >
> > > 64 128 256 500 800 1000 1500
> > > vm 0.23 0.42 0.75 1.3 2.3 2.7 3.9
> > > host 3.6 6.35 8.3 9.5 9.7 9.8 9.82
> > >
> > > I have to improve it dramatically.
> > >
> > >
> > >
> > > On Mon, Dec 26, 2016 at 2:52 AM Hu, Xuekun 
> wrote:
> > >
> > > Searching “hugepages” in https://libvirt.org/formatdomain.html
> > >
> > >
> > >
> > > If you are looking for to measure in and out packets through host,
> maybe
> > > you can look at vhost/virtio interface also.
> > >
> > >
> > >
> > > After your testing, if you can report the performace out with macvtap,
> > > that also helps us. J
> > >
> > >
> > >
> > >
> > >
> > > *From:* edgar helmut [mailto:helmut.edgar...@gmail.com]
> > > *Sent:* Saturday, December 24, 2016 11:53 PM
> > >
> > >
> > > *To:* Hu, Xuekun 
> > > *Cc:* Wiles, Keith ; users@dpdk.org
> > > *Subject:* Re: [dpdk-users] Dpdk poor performance on virtual machine
> > >
> > >
> > >
> > > any idea how to reserve hugepages for a guest (and not
> > > transparent/anonymous hugepages) ?
> > >
> > > i am using libvirt and any backing method I am trying results with
> > > anonymous hugepage.
> > >
> > > disabling the transparent hugepages resulted without any hugepages.
> > >
> > > Thanks
> > >
> > >
> > >
> > > On Sat, Dec 24, 2016 at 10:06 AM edgar helmut <
> helmut.edgar...@gmail.com>
> > > wrote:
> > >
> > > I am looking for a mean to measure in and out packets to and from the
> vm
> > > (without asking the vm itself). While pure passthrough doesn't expose
> an
> > > interface to query for in/out pkts the macvtap exposes such an
> interface.
> > >
> > > As for the anonymous hugepages I was looking for a more flexible method
> > > and I assumed there is no much difference.
> > >
> > > I will make the test with reserved hugepages.
> > >
> > > However is there any knowledge about macvtap performance issues when
> > > delivering 5-6 gbps?
> > >
> > >
> > >
> > > Thanks
> > >
> > >
> > >
> > >
> > >
> > > On 24 Dec 2016 9:06 AM, "Hu, Xuekun"  wrote:
> > >
> > > Now your setup has a new thing, “macvtap”. I don’t know what’s the
> > > performance of using macvtap. I only know it has much worse perf than
> the
> > > “real” pci pass-through.
> > >
> > >
> > >
> > > I also don’t know why you select such config for your setup, anonymous
> > > huge pages and macvtap. Any specific purpose?
> > >
> > >
> > >
> > > I think you should get a baseline first, then to get how much perf
> dropped
> > > if using anonymous hugepages or macvtap。
> > >
> > > 1.  Baseline: real hugepage + real pci pass-through
> > >
> > > 2.  Anon hugepages vs hugepages
> > >
> > > 3.  Real pci pass-through vs. macvtap
> > >
> > >
> > >
> > > *From:* edgar helmut [mailto:helmut.edgar...@gmail.com]
> > > *Sent:* Saturday, December 24, 2016 3:23 AM
> > > *To:* Hu, Xuekun 
> > > *Cc:* Wiles, Keith ; users@dpdk.org
> > >
> > >
> > > *Subject:* Re: [dpdk-users] Dpdk poor performance on virtual machine
> > >
> > >
> > >
> > > Hello,
> > >
> > > I changed the