[dpdk-dev] DPDK 1.6 and Cisco VIC-NIC pmd fails to initialize with VMWare ESX 5.5

[Patel, Rashmin N]

What is the error code it failed with?

There is no dependency between ESXi and DPDK versions as such. But depending 
your VM settings, it may pop up some issues related to memory assigned to VM.

When you use NIC in Hypervisor and feed packets to VM via Vmxnet3, the guest OS 
can use Vmxnet3-pmd and that's the sole requirement. Guest OS or DPDK API in VM 
are unaware and independent of the physical NIC beneath.
When you use NIC in VM via Passthrough (or device direct assignment,) the guest 
OS can use ixgbe-pmd/igb-pmd/xyz-pmd to control them for Rx/Tx operations. ESXi 
can't look into such device operations like it can't see packets coming in or 
going out.
When you enable SRIOV on a NIC, you get a PF and multiple VFs and if you use a 
NIC-Virtual Function via passthrough, the guest OS can use respective pmd 
(ixgbevf/igbvf/xyzvf) in DPDK API to control it for Rx/Tx ops. ESXi control the 
Physical Function through a respective driver (ixgbe/igb/xyz provided with 
ESXi) DPDK API doesn't have support to run in ESXi to control that PF.

Thanks,
Rashmin

-Original Message-
From: dev [mailto:dev-boun...@dpdk.org] On Behalf Of Karmarkar Suyash
Sent: Wednesday, November 18, 2015 8:34 PM
To: dev at dpdk.org
Subject: [dpdk-dev] DPDK 1.6 and Cisco VIC-NIC pmd fails to initialize with 
VMWare ESX 5.5

Hello All,

I am trying some experiments with DPDK 1.6 and Cisco NIC - (:06:00.0 
Network controller: Cisco Systems Inc. Cisco VIC Ethernet NIC [vmnic0]), on the 
guest I am using VMXNET3 as interface/pmd and one weird behavior that is see is 
with VMWare ESX 5.5 DPDK is not able to initialize the NIC and I get the error 
rte_eth_dev_start failed with error code as, but same NIC and with DPDK 1.6 
version works fine with VMWare ESX 6.0 and with VMXNET3 PMD.

Two questions:

1. Is there any dependency with VMWare version ESX (5.0/5.1/6.0) with DPDK 
Release version (1.6/1.7/2.x)? If yes can you please let me know?

2. If we are not using the NIC in pass-through mode and we are using it via 
Hypervisor - vmxnet3 or virtio pmd is there still a dependency for the list of 
NIC's supported by DPDK?

Regards
Suyash

[dpdk-dev] DPDK 2.2 roadmap

[Patel, Rashmin N]

There were two line items on the 2.2 roadmap: Xen Driver and Hyper-V Driver. 
Can you provide some more details?

Thanks,
Rashmin

-Original Message-
From: dev [mailto:dev-boun...@dpdk.org] On Behalf Of Thomas Monjalon
Sent: Wednesday, September 09, 2015 1:45 AM
To: dev at dpdk.org
Subject: [dpdk-dev] DPDK 2.2 roadmap

Hello,

The new features for the release 2.2 must be first submitted before 2nd October.
They should be integrated before 23rd October.

In order to ease cooperation and integration, it would be nice to see announces 
or technical details of planned features for 2.2 or 2.3.
Then the roadmap page will be filled accordingly:
http://dpdk.org/dev/roadmap
Generally speaking, it helps to know what you are working on and what is the 
status.

Thanks

[dpdk-dev] VMXNET3 on vmware, ping delay

[Patel, Rashmin N]

For tuning ESXi and vSwitch for latency sensitive workloads, I remember the 
following paper published by VMware: 
https://www.vmware.com/files/pdf/techpaper/VMW-Tuning-Latency-Sensitive-Workloads.pdf
 that you can try out.

The overall latency in setup (vmware and dpdk-vm using vmxnet3) remains in 
vmware-native-driver/vmkernel/vmxnet3-backend/vmx-emulation threads in ESXi. So 
you can better tune ESXi (as explained in the above white paper) and/or make 
sure that these important threads are not starving to improve latency and 
throughput in some cases of this setup.

Thanks,
Rashmin

-Original Message-
From: dev [mailto:dev-boun...@dpdk.org] On Behalf Of Matthew Hall
Sent: Thursday, June 25, 2015 8:19 AM
To: Vass, Sandor (Nokia - HU/Budapest)
Cc: dev at dpdk.org
Subject: Re: [dpdk-dev] VMXNET3 on vmware, ping delay

On Thu, Jun 25, 2015 at 09:14:53AM +, Vass, Sandor (Nokia - HU/Budapest) 
wrote:
> According to my understanding each packet should go through BR as fast 
> as possible, but it seems that the rte_eth_rx_burst retrieves packets 
> only when there are at least 2 packets on the RX queue of the NIC. At 
> least most of the times as there are cases (rarely - according to my 
> console log) when it can retrieve 1 packet also and sometimes only 3 
> packets can be retrieved...

By default DPDK is optimized for throughput not latency. Try a test with 
heavier traffic.

There is also some work going on now for DPDK interrupt-driven mode, which will 
work more like traditional Ethernet drivers instead of polling mode Ethernet 
drivers.

Though I'm not an expert on it, there is also a series of ways to optimize for 
latency, which hopefully some others could discuss... or maybe search the 
archives / web site / Intel tuning documentation.

Matthew.

[dpdk-dev] Regarding usage of vmxnet3 PMD with DPDK2.0

[Patel, Rashmin N]

That's correct. If you're using dpdk packaged vmxnet3 pmd, you should be able 
to bind it to igb_uio in Guest OS and start l2fwd/ l3fwd app in VM.

On the ESXi part, your vmxnet3 device must be connected to one of the vSwitches 
with Uplink port, so you don't need to configure anything else, I believe.
Just make sure the packets coming in the Uplink port carries your vmxnet3 
device's mac-vlan and they'll hit your device and the VM, ultimately ending up 
in your dpdk thread polling on vmxnet3 device in your app.

Thanks,
Rashmin

On Jun 24, 2015 9:54 PM, Puneet Singh  wrote:
I am migrating from using DPDK1.6r2 and the external vmxnet3 user map
driver to DPDK2.0.
I believe with DPDK2.0, the vmxnet3 PMD is builtin and I don?t need to
use the external driver.
I have been able to build the DPDK2.0 and I see that
CONFIG_RTE_LIBRTE_VMXNET3_PMD is set to y in the config.

Question is, what should be the steps that I follow on the ESXi node
in my virtual machine to detect the NIC with the above driver (I do
have vmxnet3 NIC there)
Should I load the uio and take over the NIC the usual way with igb_uio
and then run my application or are there any other steps to be
followed.

Regards
-Puneet

[dpdk-dev] Debugging SR-IOV related packet flow problem with

[Patel, Rashmin N]

You can pass uio device to docker container with below command line, or you can 
mount the volume /dev to container to allow all devices.

docker run -it --rm --privileged --device=/dev/uio0:/dev/uio0 
--device=/dev/uio1:/dev/uio1 -v /mnt:/mnt fedora20-dpdk-2 /bin/bash

-Original Message-
From: dev [mailto:dev-boun...@dpdk.org] On Behalf Of Mohit Saxena
Sent: Wednesday, June 17, 2015 1:41 AM
To: dev
Subject: Re: [dpdk-dev] Debugging SR-IOV related packet flow problem with

Hi,

Can you please tell me the steps for allocating uio to docker?


-- 
Thanks & Regards,
Mohit Prakash Saxena
Citrix R & D India Pvt. Ltd.
Mb: +91 90368 98569 <#UNIQUE_ID_SafeHtmlFilter_>
Email: mohit.saxena at citrix.com 

[dpdk-dev] DPDK on EC2?

[Patel, Rashmin N]

Just curious - I didn't understand the usage model here. In the Amazon EC2 
instance (a VM,) do you want to run a DPDK application? Where do you intend to 
run OVS? 

As far I remember, OVS has a datapath in Host kernel and you won't be given 
Host access by AWS. And I'm not sure if Amazon provides an OVS instance, but in 
case if they don't have an offering, you have to put your OVS installation in 
another VM (virtualizing network function) and bounce the traffic back to your 
VM(s.)

As far as Amazon Advanced Networking is concerned, you get a dedicated slice 
(an SRIOV Virtual Function) of a NIC in your VM, where a Physical Function 
Driver running in the Host (Amazon VMM-Xen I think) controls all the filtering 
configuration privileges.

Thanks,
Rashmin

-Original Message-
From: dev [mailto:dev-boun...@dpdk.org] On Behalf Of Andre Richter
Sent: Monday, March 16, 2015 2:27 PM
To: Jeff Wang; dev at dpdk.org
Subject: Re: [dpdk-dev] DPDK on EC2?

Be sure to get an instance with SR-IOV, which is "enhanced networking" in 
Amazon speak.

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking.html

Cheers Andre
Jeff Wang  schrieb am Mo., 16. M?rz 2015 um 21:08:

> Hi,
>
> I'm trying to deploy DPDK and ovs on an AWS EC2 instance which is 
> running CentOS 7.
>
> Has anybody had tried to do this? How to bind a NIC to dpdk while it 
> shows no NICs when I do lspci?
>
> Thanks!
>

[dpdk-dev] vmware vmxnet3-usermap AND DPDK VMXNET3 PMD

[Patel, Rashmin N]

Please find comments in-lined.

Thanks,
RP

From: Aziz Hajee [mailto:a...@saisei.com]
Sent: Monday, November 10, 2014 6:00 PM
To: Patel, Rashmin N
Cc: dev at dpdk.org
Subject: Re: [dpdk-dev] vmware vmxnet3-usermap AND DPDK VMXNET3 PMD

Rashmin,
Since I do need the jumbo, I use the vmxnet3-plugin you described, i.e.
(1)
sudo insmod ./vmxnet3-usermap.ko enable_shm=2,2 num_rqs=1,1 num_rxds=2048 
num_txds=2048
and (2) when running the application, use in the args list:
"-d", "librte_pmd_vmxnet3.so"
Does the above two piece mean vmxnet3-plugin
[RP] that?s correct
I do see my vmxnet3 device from the dump,rte_eal_pci_dump();
but the 'nb_ports' in DPDK never gets incremented rte_eth_dev_count() returns 
zero.
so all the other api fails, if (port_id >= nb_ports) {
PMD_DEBUG_TRACE("Invalid port_id=%d\n", port_id);
return;
}

:03:00.0 - vendor:15ad device:7b0
   d2404000 1000
   d2403000 1000
   d240 2000
    
    
    
   d440 0001
:0b:00.0 - vendor:15ad device:7b0
   d2504000 1000
   d2503000 1000
   d250 2000
    
    
    
   d450 0001
DPDK: No Ethernet ports (rte_eth_dev_count() returns zero)
PMD: rte_eth_dev_info_get: Invalid port_id=0
PMD: rte_eth_dev_configure: Invalid port_id=0
PMD: rte_eth_dev_info_get: Invalid port_id=0
PMD: rte_eth_dev_configure: Invalid port_id=0
So when using not using DPDK PMD for VMXNET3, what am i missing, for the the 
DPDK to know the nb_ports,
How will the rte_eth_dev_start(portid) in DPDK library know  ?

rte_pmd_init_all() will not have the init the Intel DPDK PMD , 
RTE_LIBRTE_VMXNET3_PMD = n in config.
[RP] If you?re using the vmxnet3-plugin, you should keep RTE_LIBRTE_VMXNET3_PMD 
= n in config, and link the shared library with its headers, it should work. I 
have tried it once a long back.

#ifdef RTE_LIBRTE_VMXNET3_PMD
if ((ret = rte_vmxnet3_pmd_init()) != 0) {
RTE_LOG(ERR, PMD, "Cannot init vmxnet3 PMD\n");
return (ret);
}
If I make RTE_LIBRTE_VMXNET3_PMD = y, then I am using the Intel DPDK PMD and no 
jumbo.
[RP] Yes, I understood that part. We need to support jumbo frames in in-tree 
version of VMXNET3-PMD, we?ll merge all soon, we can discuss in the community 
conf. call so please do attend the next one on Nov 18 and we can raise concerns 
there

Thanks,
aziz

On Fri, Nov 7, 2014 at 8:53 AM, Patel, Rashmin N mailto:rashmin.n.patel at intel.com>> wrote:
Hi Aziz,

Yes, you're right DPDK VMXNET3-PMD in /lib/librte_pmd_vmxnet3 does not support 
mbuf chaining today. But it's a standalone bsd driver just like any other pmd 
in that directory, it does not need vmxnet3-usermap.ko module.

Now there is another vmxnet3 solution in a separate branch as a plugin, which 
must have vmxnet3-usermap.ko linux module(1), and a user space interface 
piece(2) to tie it to any DPDK application in the main branch. (1) and (2) 
makes the solution which is known as vmxnet3-plugin. It's been there for a long 
time just like virtio-plugin, I don't know who uses it, but community can 
*reply* here if there is still any need of a separate solution that way.

I'm in favor of consolidating all those version into one elegant solution by 
grabbing best features from all of them and maintain one copy. I'm sure that 
developers contributing from VMware would also support that idea because then 
it makes easy to maintain and debug and bug fix and most importantly avoid such 
confusion in future.

Thanks,
Rashmin

-Original Message-
From: dev [mailto:dev-bounces at dpdk.org<mailto:dev-boun...@dpdk.org>] On 
Behalf Of Aziz Hajee
Sent: Thursday, November 06, 2014 5:47 PM
To: dev at dpdk.org<mailto:dev at dpdk.org>
Subject: [dpdk-dev] vmware vmxnet3-usermap AND DPDK VMXNET3 PMD

I am using the dpdk1.6.0r1
I could not find a complete clarification, sorry if missed.
VMXNET3 PMD

I have enabled the VMXNET3 PMD  in the dpdk.
 # Compile burst-oriented VMXNET3 PMD driver  #

CONFIG_RTE_LIBRTE_VMXNET3_PMD=y
CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_INIT=y
CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_RX=n
CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_TX=n
CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_TX_FREE=n
The Intel DPDK VMXNET3 PMD driver does not support mbuf chaining, and I have to 
set NOMULTSEGS for the vmxnet3 interface init to succeed.
tx_conf.txq_flags =  ETH_TXQ_FLAGS_NOMULTSEGS Is there a later version of DPDK 
that supports multiseg for the dpdk
VMXNET3 PMD.

vmware vmxnet3-usermap AND  DPDK VMXNET3 PMD 
=
Is the vmxnet3-usermap.

[dpdk-dev] vmware vmxnet3-usermap AND DPDK VMXNET3 PMD

[Patel, Rashmin N]

I wasn't sure about why anyone would "avoid using UIO just for Virtio even 
though other devices need uio/vfio in DPDK." I mean I haven't seen anyone using 
that way and performance wise they both are similar.

But anyways here are the major reasons we had Intel versions:
1. Intel Virtio-PMD version 
- was elegantly integrated in DPDK code tree and to keep consistency it 
uses UIO/VFIO framework just like any other PMDs in DPDK/lib 

2. Intel VMXNET3-PMD 
- does not depend on any other module (i.e. vmxnet3 plugin depends on 
vmxnet3-usermap.ko module)- was elegantly integrated in DPDK code tree 
and to keep consistency it uses UIO/VFIO framework just like any other PMDs in 
DPDK/lib
- is the version VMware contributes today as they think it's a better 
idea and they would provide ESXi support for that if they need to

Thanks,
Rashmin


-Original Message-
From: Thomas Monjalon [mailto:thomas.monja...@6wind.com] 
Sent: Sunday, November 09, 2014 4:18 PM
To: dev at dpdk.org
Cc: Patel, Rashmin N; Aziz Hajee
Subject: Re: [dpdk-dev] vmware vmxnet3-usermap AND DPDK VMXNET3 PMD

Hi,

2014-11-07 16:53, Patel, Rashmin N:
> Yes, you're right DPDK VMXNET3-PMD in /lib/librte_pmd_vmxnet3 does not 
> support mbuf chaining today. But it's a standalone bsd driver just 
> like any other pmd in that directory, it does not need vmxnet3-usermap.ko 
> module.
> 
> Now there is another vmxnet3 solution in a separate branch as a 
> plugin, which must have vmxnet3-usermap.ko linux module(1), and a user 
> space interface piece(2) to tie it to any DPDK application in the main branch.
> (1) and (2) makes the solution which is known as vmxnet3-plugin. It's 
> been there for a long time just like virtio-plugin, I don't know who 
> uses it, but community can *reply* here if there is still any need of 
> a separate solution that way.

Coming back to last year, 6WIND developped some PMDs for virtio and vmxnet3.
Later, Intel developped their own version using the uio framework.
The versions in the main repository are the Intel ones, whereas the original 
ones from 6WIND are released as extensions.
For completeness, it must be noted that Brocade worked on their own approach of 
vmxnet3 and contributed to virtio PMD.
It's now time to merge all these implementations.

The 6WIND implementations show that it's possible to avoid the uio framework.
The virtio-net-pmd use port access granted by iopl().
The vmxnet3-usermap reuse the VMware's kernel module with a special mode for 
memory mapping. It was a pre-bifurcated logic.

> I'm in favor of consolidating all those version into one elegant 
> solution by grabbing best features from all of them and maintain one 
> copy. I'm sure that developers contributing from VMware would also 
> support that idea because then it makes easy to maintain and debug and 
> bug fix and most importantly avoid such confusion in future.


> -Original Message-
> From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Aziz Hajee
> Sent: Thursday, November 06, 2014 5:47 PM
> To: dev at dpdk.org
> Subject: [dpdk-dev] vmware vmxnet3-usermap AND DPDK VMXNET3 PMD
> 
> I am using the dpdk1.6.0r1
> I could not find a complete clarification, sorry if missed.
> VMXNET3 PMD
> 
> I have enabled the VMXNET3 PMD  in the dpdk.
>  # Compile burst-oriented VMXNET3 PMD driver  #
> 
> CONFIG_RTE_LIBRTE_VMXNET3_PMD=y
> CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_INIT=y
> CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_RX=n
> CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_TX=n
> CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_TX_FREE=n
> The Intel DPDK VMXNET3 PMD driver does not support mbuf chaining, and I have 
> to set NOMULTSEGS for the vmxnet3 interface init to succeed.
> tx_conf.txq_flags =  ETH_TXQ_FLAGS_NOMULTSEGS Is there a later version 
> of DPDK that supports multiseg for the dpdk
> VMXNET3 PMD.
> 
> vmware vmxnet3-usermap AND  DPDK VMXNET3 PMD 
> =
> Is the vmxnet3-usermap.ko module driver also needed ? (appears that I need, 
> otherwise the eal initialise fails.
> sudo insmod ./vmxnet3-usermap.ko enable_shm=2,2 num_rqs=1,1 
> num_rxds=2048
> num_txds=2048
> 
> I do not understand if VMXNET3 PMD is there, what is the purpose of 
> /vmxnet3-usermap.ko/vmxnet3-usermap.ko ?
> 
> From some responses i saw that the following ifdef RTE_EAL_UNBIND_PORTS is 
> also need to be removed in lib/librte_pmd_vmxnet3/vmxnet3_ethdev.c ?
>  {
>  .name = "rte_vmxnet3_pmd",
>  .id_table = pci_id_vmxnet3_map, -#ifdef RTE_EAL_UNBIND_PORTS
> +// #ifdef RTE_EAL_UNBIND_PORTS
>  .drv_flags = RTE_PCI_DRV_NEED_IGB_UIO, -#endif
> +// #endif
>  },
>  .eth_dev_init = eth_vmxnet3_dev_init,
>  .dev_private_size = sizeof(struct vmxnet3_adapter),
> 
> thanks,
> -aziz

[dpdk-dev] vmware vmxnet3-usermap AND DPDK VMXNET3 PMD

[Patel, Rashmin N]

Hi Aziz,

Yes, you're right DPDK VMXNET3-PMD in /lib/librte_pmd_vmxnet3 does not support 
mbuf chaining today. But it's a standalone bsd driver just like any other pmd 
in that directory, it does not need vmxnet3-usermap.ko module.

Now there is another vmxnet3 solution in a separate branch as a plugin, which 
must have vmxnet3-usermap.ko linux module(1), and a user space interface 
piece(2) to tie it to any DPDK application in the main branch. (1) and (2) 
makes the solution which is known as vmxnet3-plugin. It's been there for a long 
time just like virtio-plugin, I don't know who uses it, but community can 
*reply* here if there is still any need of a separate solution that way.

I'm in favor of consolidating all those version into one elegant solution by 
grabbing best features from all of them and maintain one copy. I'm sure that 
developers contributing from VMware would also support that idea because then 
it makes easy to maintain and debug and bug fix and most importantly avoid such 
confusion in future.

Thanks,
Rashmin

-Original Message-
From: dev [mailto:dev-boun...@dpdk.org] On Behalf Of Aziz Hajee
Sent: Thursday, November 06, 2014 5:47 PM
To: dev at dpdk.org
Subject: [dpdk-dev] vmware vmxnet3-usermap AND DPDK VMXNET3 PMD

I am using the dpdk1.6.0r1
I could not find a complete clarification, sorry if missed.
VMXNET3 PMD

I have enabled the VMXNET3 PMD  in the dpdk.
 # Compile burst-oriented VMXNET3 PMD driver  #

CONFIG_RTE_LIBRTE_VMXNET3_PMD=y
CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_INIT=y
CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_RX=n
CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_TX=n
CONFIG_RTE_LIBRTE_VMXNET3_DEBUG_TX_FREE=n
The Intel DPDK VMXNET3 PMD driver does not support mbuf chaining, and I have to 
set NOMULTSEGS for the vmxnet3 interface init to succeed.
tx_conf.txq_flags =  ETH_TXQ_FLAGS_NOMULTSEGS Is there a later version of DPDK 
that supports multiseg for the dpdk
VMXNET3 PMD.

vmware vmxnet3-usermap AND  DPDK VMXNET3 PMD 
=
Is the vmxnet3-usermap.ko module driver also needed ? (appears that I need, 
otherwise the eal initialise fails.
sudo insmod ./vmxnet3-usermap.ko enable_shm=2,2 num_rqs=1,1 num_rxds=2048
num_txds=2048

I do not understand if VMXNET3 PMD is there, what is the purpose of 
/vmxnet3-usermap.ko/vmxnet3-usermap.ko ?

[dpdk-dev] vmxnet3 pmd dev restart

[Patel, Rashmin N]

Waterman/Navakanth, we've got some patches posted for the same by Yong Wang at 
VMware. I haven't got the chance to look at it but if you can validate it, it'd 
be great.

Thanks,
Rashmin

-Original Message-
From: Navakanth M [mailto:navakanth...@gmail.com] 
Sent: Sunday, October 12, 2014 8:07 PM
To: Patel, Rashmin N; Cao, Waterman
Cc: stephen at networkplumber.org; dev at dpdk.org; Jiajia, SunX
Subject: Re: vmxnet3 pmd dev restart

Hi Rashmin

I have tried the memset change but still I am facing the problem which I 
pointed out earlier. After restart, packets are not being received in 
vmxnet3_recv_pkts().
I have also observed PANIC in vmxnet3_tq_tx_complete() after couple of stop and 
start operations.

PANIC in vmxnet3_tq_tx_complete():
EOP desc does not point to a valid mbuf15:
[/lib64/libc.so.6(clone+0x6d) [0x7fd60354c52d]]
1: [/mswitch/bin/sos.shumway.elf(rte_dump_stack+0x23) [0x463313]]
2: [/mswitch/bin/sos.shumway.elf(__rte_panic+0xc1) [0x447ae8]]
3: [/mswitch/bin/sos.shumway.elf(vmxnet3_xmit_pkts+0x382) [0x4f4f22]]

Thanks
Navakanth

On Fri, Oct 10, 2014 at 8:39 AM, Cao, Waterman  
wrote:
> Hi Rashmin,
>
>  We found similar issue when we start/stop vmnet dev several time. (> 
> 3 times)  It happens kernel panic, and sometimes kernel will occur core dump.
>  Let me know if you want to submit patch to fix it.
>
> Thanks
> Waterman
>
> -----Original Message-
>>From: Patel, Rashmin N
>>Sent: Friday, October 10, 2014 6:07 AM
>>To: Navakanth M; stephen at networkplumber.org; Cao, Waterman
>>Cc: dev at dpdk.org
>>Subject: RE: vmxnet3 pmd dev restart
>>
>>I just quickly looked into the code and instead of releasing memory or simply 
>>set it to NULL (patch:
>> http://thread.gmane.org/gmane.comp.networking.dpdk.devel/4683), you can zero 
>> it out and it should work perfectly, you can give it a quick try.
>>
>>//rte_free(ring->buf_info);
>>memset(ring->buf_info, 0x0, ring->size*sizeof(vmxnet3_buf_info_t));
>>
>>This will not free the memory from heap but just wipe it out to 0x0, provided 
>>that we freed all the mbuf(s) pointed by each buf_info->m pointers. Hence you 
>>won't need to reallocate it when you start device after this stop.
>>
>>Thanks,
>>Rashmin
>>
>>-Original Message-
>>From: Navakanth M [mailto:navakanthdev at gmail.com]
>>Sent: Wednesday, October 08, 2014 10:11 PM
>>To: stephen at networkplumber.org; Patel, Rashmin N; Cao, Waterman
>>Cc: dev at dpdk.org
>>Subject: Re: vmxnet3 pmd dev restart
>>
>>I had tried with Stephen's patch but after stop is done and when we 
>>call start it crashed at vmxnet3_dev_start()->
>>vmxnet3_dev_rxtx_init()->vmxnet3_post_rx_bufs() as buf_info is freed 
>>and is not allocated again. buf_info is allocated in
>>vmxnet3_dev_rx_queue_setup() which would be called once at the initialization 
>>only.
>>I also tried not freeing buf_info in stop but then i see different 
>>issue after start, packets are not received due to check while 
>>(rcd->gen == rxq->comp_ring.gen) { in vmxnet3_recv_pkts()
>>
>>Waterman, Have you got chance to test stop and start of vmnet dev if so did 
>>you notice any issue similar to this?
>>
>>Thanks
>>Navakanth
>>
>>On Thu, Oct 9, 2014 at 12:46 AM, Patel, Rashmin N >intel.com> wrote:
>>> Yes I had a local copy working with couple of lines fix. But someone else, 
>>> I think Stephen added a fix patch for the same, and I assume if it's been 
>>> merged, should be working, so did not follow up later.
>>>
>>> I don't have a VMware setup handy at moment but I think Waterman would have 
>>> more information about testing that patch if he has found any issue with it.
>>>
>>> Thanks,
>>> Rashmin
>>>
>>> -Original Message-
>>> From: Navakanth M [mailto:navakanthdev at gmail.com]
>>> Sent: Wednesday, October 08, 2014 4:14 AM
>>> To: dev at dpdk.org; Patel, Rashmin N
>>> Subject: Re: vmxnet3 pmd dev restart
>>>
>>> Hi Rashmin
>>>
>>> I have come across your reply in following post that you have worked on 
>>> this problem and would submit the patch for it.
>>> Can you please share information on the changes you worked on or patch log 
>>> if you had submitted any for it?
>>> http://thread.gmane.org/gmane.comp.networking.dpdk.devel/4683
>>>
>>> Thanks
>>> Navakanth
>>>
>>> On Tue, Sep 30, 2014 at 1:44 PM, Navakanth M  
>>> wrote:
>>>> Hi
>>>>
>>>> I am using DPDKv1.7.0 

[dpdk-dev] vmxnet3 pmd dev restart

[Patel, Rashmin N]

I just quickly looked into the code and instead of releasing memory or simply 
set it to NULL (patch: 
 http://thread.gmane.org/gmane.comp.networking.dpdk.devel/4683), you can zero 
it out and it should work perfectly, you can give it a quick try.

//rte_free(ring->buf_info);
memset(ring->buf_info, 0x0, ring->size*sizeof(vmxnet3_buf_info_t));

This will not free the memory from heap but just wipe it out to 0x0, provided 
that we freed all the mbuf(s) pointed by each buf_info->m pointers. Hence you 
won't need to reallocate it when you start device after this stop.

Thanks,
Rashmin

-Original Message-
From: Navakanth M [mailto:navakanth...@gmail.com] 
Sent: Wednesday, October 08, 2014 10:11 PM
To: stephen at networkplumber.org; Patel, Rashmin N; Cao, Waterman
Cc: dev at dpdk.org
Subject: Re: vmxnet3 pmd dev restart

I had tried with Stephen's patch but after stop is done and when we call start 
it crashed at vmxnet3_dev_start()->
vmxnet3_dev_rxtx_init()->vmxnet3_post_rx_bufs() as buf_info is freed and is not 
allocated again. buf_info is allocated in
vmxnet3_dev_rx_queue_setup() which would be called once at the initialization 
only.
I also tried not freeing buf_info in stop but then i see different issue after 
start, packets are not received due to check while (rcd->gen == 
rxq->comp_ring.gen) { in vmxnet3_recv_pkts()

Waterman, Have you got chance to test stop and start of vmnet dev if so did you 
notice any issue similar to this?

Thanks
Navakanth

On Thu, Oct 9, 2014 at 12:46 AM, Patel, Rashmin N  wrote:
> Yes I had a local copy working with couple of lines fix. But someone else, I 
> think Stephen added a fix patch for the same, and I assume if it's been 
> merged, should be working, so did not follow up later.
>
> I don't have a VMware setup handy at moment but I think Waterman would have 
> more information about testing that patch if he has found any issue with it.
>
> Thanks,
> Rashmin
>
> -Original Message-
> From: Navakanth M [mailto:navakanthdev at gmail.com]
> Sent: Wednesday, October 08, 2014 4:14 AM
> To: dev at dpdk.org; Patel, Rashmin N
> Subject: Re: vmxnet3 pmd dev restart
>
> Hi Rashmin
>
> I have come across your reply in following post that you have worked on this 
> problem and would submit the patch for it.
> Can you please share information on the changes you worked on or patch log if 
> you had submitted any for it?
> http://thread.gmane.org/gmane.comp.networking.dpdk.devel/4683
>
> Thanks
> Navakanth
>
> On Tue, Sep 30, 2014 at 1:44 PM, Navakanth M  
> wrote:
>> Hi
>>
>> I am using DPDKv1.7.0 running on Vmware Esxi 5.1 and am trying to 
>> reset the port which uses pmd_vmnet3 library functions from below 
>> function calls.
>> rte_eth_dev_stop
>> rte_eth_dev_start
>>
>> Doing this, i face panic while rte_free(ring->buf_info) in 
>> Vmxnet3_cmd_ring_release().
>> I have gone through following thread but the patch mentioned didn't 
>> help rather it crashed in start function while accessing buf_info in 
>> vmxnet3_post_rx_bufs. I see this buf_info is allocated in queue setup 
>> functions which are called at initialization.
>> http://thread.gmane.org/gmane.comp.networking.dpdk.devel/4683
>>
>> I tried not freeing it and then rx packets are not received due to 
>> mismatch in while (rcd->gen == rxq->comp_ring.gen) in
>> vmxnet3_recv_pkts()
>>
>> To reset the device port, is this the right way what i am doing?
>> Or do I have to call vmxnet3_dev_tx_queue_setup()
>> vmxnet3_dev_rx_queue_setup() once stop is called? I have checked 
>> recent patches and threads but did not get much information on this.
>>
>> Thanks
>> Navakanth

[dpdk-dev] vmxnet3 pmd dev restart

[Patel, Rashmin N]

Yes I had a local copy working with couple of lines fix. But someone else, I 
think Stephen added a fix patch for the same, and I assume if it's been merged, 
should be working, so did not follow up later. 

I don't have a VMware setup handy at moment but I think Waterman would have 
more information about testing that patch if he has found any issue with it.

Thanks,
Rashmin

-Original Message-
From: Navakanth M [mailto:navakanth...@gmail.com] 
Sent: Wednesday, October 08, 2014 4:14 AM
To: dev at dpdk.org; Patel, Rashmin N
Subject: Re: vmxnet3 pmd dev restart

Hi Rashmin

I have come across your reply in following post that you have worked on this 
problem and would submit the patch for it.
Can you please share information on the changes you worked on or patch log if 
you had submitted any for it?
http://thread.gmane.org/gmane.comp.networking.dpdk.devel/4683

Thanks
Navakanth

On Tue, Sep 30, 2014 at 1:44 PM, Navakanth M  wrote:
> Hi
>
> I am using DPDKv1.7.0 running on Vmware Esxi 5.1 and am trying to 
> reset the port which uses pmd_vmnet3 library functions from below 
> function calls.
> rte_eth_dev_stop
> rte_eth_dev_start
>
> Doing this, i face panic while rte_free(ring->buf_info) in 
> Vmxnet3_cmd_ring_release().
> I have gone through following thread but the patch mentioned didn't 
> help rather it crashed in start function while accessing buf_info in 
> vmxnet3_post_rx_bufs. I see this buf_info is allocated in queue setup 
> functions which are called at initialization.
> http://thread.gmane.org/gmane.comp.networking.dpdk.devel/4683
>
> I tried not freeing it and then rx packets are not received due to 
> mismatch in while (rcd->gen == rxq->comp_ring.gen) in 
> vmxnet3_recv_pkts()
>
> To reset the device port, is this the right way what i am doing?
> Or do I have to call vmxnet3_dev_tx_queue_setup()
> vmxnet3_dev_rx_queue_setup() once stop is called? I have checked 
> recent patches and threads but did not get much information on this.
>
> Thanks
> Navakanth

[dpdk-dev] Hi all, does Amazon VMs supported DPDK or not?

[Patel, Rashmin N]

You answered the question but I was looking for more details. The term you 
called "ENI" is new to me, so what kind of device is it? It must be one of this:
1. Emulated Device (I suspect this is the one; because DPDK API works without 
any change)
2. Para-virtual Device
3. Directly Assigned Device
4. Fully customized Device (i.e shared memory with GrantTable mechanism)
   or
5. A generic wrapper device (covering all of the above devices)

Thanks,
Rashmin

-Original Message-
From: Saha, Avik (AWS) [mailto:aviks...@amazon.com] 
Sent: Tuesday, September 30, 2014 10:49 AM
To: Patel, Rashmin N; Wang, Shawn; Dong, Binghua; dev at dpdk.org
Subject: RE: Hi all, does Amazon VMs supported DPDK or not?

I have only experimented with C3.8xlarge instances with DPDK. You have to 
attach at least 2 ENIs to the instances since one of them would be taken over 
by DPDK. Based on the size, you would be able to attach 8, 16  or 32 ENIs to 
the instances (these will be visible as ethn devices of ifconfig)

Let me know if this helps (or I am not getting the question :) )

Avik

-Original Message-----
From: Patel, Rashmin N [mailto:rashmin.n.pa...@intel.com] 
Sent: Monday, September 29, 2014 1:54 PM
To: Wang, Shawn; Dong, Binghua; dev at dpdk.org; Saha, Avik (AWS)
Subject: RE: Hi all, does Amazon VMs supported DPDK or not?

Hi Shawn,

Which network interface is visible to the VM? I mean which is the virtual 
ethernet port is used in Amazon-VM-DPDK app? And what all interfaces are 
offered based on the VM size and requirements?

Thanks,
Rashmin

-Original Message-
From: Wang, Shawn [mailto:xing...@amazon.com] 
Sent: Monday, September 29, 2014 1:50 PM
To: Dong, Binghua; Patel, Rashmin N; dev at dpdk.org; Saha, Avik (AWS)
Subject: RE: Hi all, does Amazon VMs supported DPDK or not?

Yes, you can.

>From my colleague, Saha, Avik, they are running  IntelDPDK 1.7 on c3.8xlarges.

Thanks.

From: dev [dev-bounces at dpdk.org] on behalf of Dong, Binghua 
[binghua.d...@intel.com]
Sent: Saturday, September 27, 2014 10:05 PM
To: Patel, Rashmin N; dev at dpdk.org
Subject: Re: [dpdk-dev] Hi all,  does Amazon VMs supported DPDK or not?

Hi Patel,

The customer consider that deploy DPDK application in Amazon VMs is very 
flexible and very easy global site deployment:

such as: they only need to buy a 2 lcores VM if a site only need 200Mbps 
throughput;   buy one 4 lcores VM if the throughput is 400Mbps;

the can buy different Amazon site VMs in US, German... for lower access latency;

-Original Message-
From: Patel, Rashmin N
Sent: Saturday, September 27, 2014 12:41 AM
To: Dong, Binghua; dev at dpdk.org
Subject: RE: Hi all, does Amazon VMs supported DPDK or not?

It really depends on the devices offered in the VM. If direct device assignment 
is not provided to a VM or if the node hypervisor doesn't have an optimized 
para-virtual interface to a VM, I don't see any benefit using DPDK in VMs.

-Original Message-
From: dev [mailto:dev-boun...@dpdk.org] On Behalf Of Dong, Binghua
Sent: Friday, September 26, 2014 5:47 AM
To: dev at dpdk.org
Subject: [dpdk-dev] Hi all, does Amazon VMs supported DPDK or not?

A customer plan to buy some global Amazon VMs to run their DPDK 1.3(will 
upgrade to DPDK1.6 or 1.7) based VPN applications on global sites.

Thanks a lot;

[dpdk-dev] Hi all, does Amazon VMs supported DPDK or not?

[Patel, Rashmin N]

Hi Shawn,

Which network interface is visible to the VM? I mean which is the virtual 
ethernet port is used in Amazon-VM-DPDK app? And what all interfaces are 
offered based on the VM size and requirements?

Thanks,
Rashmin

-Original Message-
From: Wang, Shawn [mailto:xing...@amazon.com] 
Sent: Monday, September 29, 2014 1:50 PM
To: Dong, Binghua; Patel, Rashmin N; dev at dpdk.org; Saha, Avik (AWS)
Subject: RE: Hi all, does Amazon VMs supported DPDK or not?

Yes, you can.

>From my colleague, Saha, Avik, they are running  IntelDPDK 1.7 on c3.8xlarges.

Thanks.

From: dev [dev-bounces at dpdk.org] on behalf of Dong, Binghua 
[binghua.d...@intel.com]
Sent: Saturday, September 27, 2014 10:05 PM
To: Patel, Rashmin N; dev at dpdk.org
Subject: Re: [dpdk-dev] Hi all,  does Amazon VMs supported DPDK or not?

Hi Patel,

The customer consider that deploy DPDK application in Amazon VMs is very 
flexible and very easy global site deployment:

such as: they only need to buy a 2 lcores VM if a site only need 200Mbps 
throughput;   buy one 4 lcores VM if the throughput is 400Mbps;

the can buy different Amazon site VMs in US, German... for lower access latency;

-Original Message-
From: Patel, Rashmin N
Sent: Saturday, September 27, 2014 12:41 AM
To: Dong, Binghua; dev at dpdk.org
Subject: RE: Hi all, does Amazon VMs supported DPDK or not?

It really depends on the devices offered in the VM. If direct device assignment 
is not provided to a VM or if the node hypervisor doesn't have an optimized 
para-virtual interface to a VM, I don't see any benefit using DPDK in VMs.

-Original Message-
From: dev [mailto:dev-boun...@dpdk.org] On Behalf Of Dong, Binghua
Sent: Friday, September 26, 2014 5:47 AM
To: dev at dpdk.org
Subject: [dpdk-dev] Hi all, does Amazon VMs supported DPDK or not?

A customer plan to buy some global Amazon VMs to run their DPDK 1.3(will 
upgrade to DPDK1.6 or 1.7) based VPN applications on global sites.

Thanks a lot;

[dpdk-dev] Hi all, does Amazon VMs supported DPDK or not?

[Patel, Rashmin N]

It really depends on the devices offered in the VM. If direct device assignment 
is not provided to a VM or if the node hypervisor doesn't have an optimized 
para-virtual interface to a VM, I don't see any benefit using DPDK in VMs.

-Original Message-
From: dev [mailto:dev-boun...@dpdk.org] On Behalf Of Dong, Binghua
Sent: Friday, September 26, 2014 5:47 AM
To: dev at dpdk.org
Subject: [dpdk-dev] Hi all, does Amazon VMs supported DPDK or not?

A customer plan to buy some global Amazon VMs to run their DPDK 1.3(will 
upgrade to DPDK1.6 or 1.7) based VPN applications on global sites.

Thanks a lot;

[dpdk-dev] reg:porting intel dpdk kit as LINC switch

[Patel, Rashmin N]

I think if he's using OTP framework, it has some interfacing for other 
languages as well, but not sure how efficient it's. In order to keep up with 
the performance, it's better to evaluate those interfaces first.

Thanks,
Rashmin

On Sep 10, 2014 1:07 AM, "Gray, Mark D"  wrote:
>
> Hi All,
> I wanted to port my intel DPDK kit as LINC switch, could you help me how can
> i do it.

* There are two open source software switches using DPDK. You could use them
as a reference.

https://github.com/openvswitch/ovs (look at lib/netdev-dpdk.c)
https://github.com/01org/dpdk-ovs

* Packet Framework in dpdk may also be a good starting point.

* At the very least you will probably want to modify the Linc dataplane to use
dpdk PMDs

One thing I am not sure about is how you would call DPDK functions from Linc as
it is written in Erlang. However, I am not familiar with Erlang.

Regards,
>
> --
> Regards,
> ANAND

[dpdk-dev] [PATCH RFC 0/3] only call iopl when necessary

[Patel, Rashmin N]

I think iopl() was added to have permission to do Port I/O for Virtio device 
handling.

-Original Message-
From: dev [mailto:dev-boun...@dpdk.org] On Behalf Of Xie, Huawei
Sent: Wednesday, August 27, 2014 2:22 AM
To: David Marchand; dev at dpdk.org
Subject: Re: [dpdk-dev] [PATCH RFC 0/3] only call iopl when necessary

Hi David:
The reason iopl is put in rte_eal_init is that we want all later created DPDK 
processes/threads inherit the iopl permission.
If you only call iopl in pmd_init, RX/TX and other threads which needs io 
permission will segmentation fault.

-huawei

> -Original Message-
> From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of David Marchand
> Sent: Tuesday, August 26, 2014 10:12 PM
> To: dev at dpdk.org
> Subject: [dpdk-dev] [PATCH RFC 0/3] only call iopl when necessary
> 
> This patch series is just a little clean up to remove the 
> unconditionnal call to iopl on linux.
> Rather than call iopl() at the eal level, let the PMD that needs it 
> call rte_eal_iopl_init().
> 
> --
> David Marchand
> 
> David Marchand (3):
>   eal/bsd: fix fd leak
>   eal: don't call rte_eal_iopl_init unconditionnally
>   eal: remove unused flags field
> 
>  lib/librte_eal/bsdapp/eal/eal.c |6 ++
>  lib/librte_eal/common/include/rte_eal.h |   17 +++--
>  lib/librte_eal/linuxapp/eal/eal.c   |   11 ---
>  lib/librte_pmd_virtio/virtio_ethdev.c   |   15 ---
>  4 files changed, 25 insertions(+), 24 deletions(-)
> 
> --
> 1.7.10.4

[dpdk-dev] overcommitting CPUs

[Patel, Rashmin N]

You're right and I've felt the same harder part of determinism with other 
hypervisors' soft switch solutions as well. I think it's worth thinking about.

Thanks,
Rashmin

On Aug 26, 2014 9:15 PM, Stephen Hemminger  
wrote:
The way to handle switch between out of poll mode is to use IRQ coalescing
parameters.
You want to hold off IRQ until there are a couple packets or a short delay.
Going out of poll mode
is harder to determine.


On Tue, Aug 26, 2014 at 9:59 AM, Zhou, Danny  wrote:

>
> > -Original Message-
> > From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Stephen Hemminger
> > Sent: Wednesday, August 27, 2014 12:39 AM
> > To: Michael Marchetti
> > Cc: dev at dpdk.org
> > Subject: Re: [dpdk-dev] overcommitting CPUs
> >
> > On Tue, 26 Aug 2014 16:27:14 +
> > "Michael  Marchetti"  wrote:
> >
> > > Hi, has there been any consideration to introduce a non-spinning
> network driver (interrupt based), for the purpose of overcommitting
> > CPUs in a virtualized environment?  This would obviously have reduced
> high-end performance but would allow for increased guest
> > density (sharing of physical CPUs) on a host.
> > >
> > > I am interested in adding support for this kind of operation, is there
> any interest in the community?
> > >
> > > Thanks,
> > >
> > > Mike.
> >
> > Better to implement a NAPI like algorithm that adapts from poll to
> interrupt.
>
> Agreed, but DPDK is currently pure poll-mode based, so unlike the NAPI'
> simple algorithm, the new heuristic algorithm should not switch from
> poll-mode to interrupt-mode immediately once there is no packet in the
> recent poll. Otherwise, mode switching will be too frequent which brings
> serious negative performance impact to DPDK.
>

[dpdk-dev] VMXNET 3

[Patel, Rashmin N]

I forgot to mention that the performance expectation I gave was for "big 
packets (>512 bytes size) consumed by multiple VMs" because ESXi is optimized 
for enterprise kind of workloads. 

-Original Message-
From: Patel, Rashmin N 
Sent: Tuesday, August 26, 2014 9:30 AM
To: 'Alex Markuze'; dev at dpdk.org
Subject: RE: [dpdk-dev] VMXNET 3

As far as I know, some of the us (developers) are submitting patches for the 
VMXNET3 driver, which is inside the DPDK package in librte_pmd_vmxnet3. I have 
used the other version for benchmarking a long back but as it was limited to 
Linux kernel and I found the librte_pmd_vmxnet3 version more elegant, I moved 
on to that one.

As far as performance is concerned, it's limited by the ESXi vSwitch and 
vmxnet3 backend code, so if you're looking for line rate at 10G, it's far down 
the road if vmxnet3 backend processing is optimized, but today you can achieve 
~55-60% of 10G in a single VM using VMXNET3, although you if you want to 
consume more of 10G, you can try increasing number of VMs.

Thanks,
Rashmin

-Original Message-
From: dev [mailto:dev-boun...@dpdk.org] On Behalf Of Alex Markuze
Sent: Tuesday, August 26, 2014 3:06 AM
To: dev at dpdk.org
Subject: [dpdk-dev] VMXNET 3

Hi
I'm looking for reasonable DPDK based solution in fully virtualised VMware 
environment.

[dpdk-dev] VMXNET 3

[Patel, Rashmin N]

As far as I know, some of the us (developers) are submitting patches for the 
VMXNET3 driver, which is inside the DPDK package in librte_pmd_vmxnet3. I have 
used the other version for benchmarking a long back but as it was limited to 
Linux kernel and I found the librte_pmd_vmxnet3 version more elegant, I moved 
on to that one.

As far as performance is concerned, it's limited by the ESXi vSwitch and 
vmxnet3 backend code, so if you're looking for line rate at 10G, it's far down 
the road if vmxnet3 backend processing is optimized, but today you can achieve 
~55-60% of 10G in a single VM using VMXNET3, although you if you want to 
consume more of 10G, you can try increasing number of VMs.

Thanks,
Rashmin

-Original Message-
From: dev [mailto:dev-boun...@dpdk.org] On Behalf Of Alex Markuze
Sent: Tuesday, August 26, 2014 3:06 AM
To: dev at dpdk.org
Subject: [dpdk-dev] VMXNET 3

Hi
I'm looking for reasonable DPDK based solution in fully virtualised VMware 
environment.

[dpdk-dev] Vmxnet3 pmd

[Patel, Rashmin N]

Hi Alex,

This is known issue with dev_stop of vmxnet3_pmd. Vmxnet3_cmd_ring_release() 
routine calls rte_free(ring->buf_info), which should be reused instead. Because 
it's not being reallocated at dev_start, otherwise reallocation is needed. I've 
a patch that works for me, I'll submit it after some testing.

Thanks,
Rashmin

-Original Message-
From: dev [mailto:dev-boun...@dpdk.org] On Behalf Of Alex Markuze
Sent: Wednesday, August 13, 2014 3:14 AM
To: dev at dpdk.org
Subject: [dpdk-dev] Vmxnet3 pmd

Hi, I Have a simple dpdk app - basically a KNI interface with the dpdk layer 
serving only as a pipeline.

This allows me to ping between vEth0 on different VM's works great with ixgbevf.
Now I moved to ESXi5.5 , Ubuntu14.4 VM (Dpdk 1.7).
When running the same code*  I've discovered that the polling doesn't retrieve 
any packets after vEth0 gents an IP. I've resolved this issue by removing the 
dev restart calls I had in the callback.

//  rte_eth_dev_stop(port_id);
//  ret = rte_eth_dev_start(port_id);

Is this a know issue? how can I report a BUG if its not.

Thanks
Alex.

*Except this line I needed to add to setup the TXQ.
tx_conf.txq_flags |= (ETH_TXQ_FLAGS_NOMULTSEGS | ETH_TXQ_FLAGS_NOOFFLOADS);

[dpdk-dev] Performance issue with vmxnet3 pmd

[Patel, Rashmin N]

According to my experiments at moment, the bottleneck lies in backend in 
hypervisor for para-virtual devices including Vmxnet3 and hence different 
front-end drivers (stock Vmxnet3 driver or Vmxnet3-PMD) would performance 
equally well, I don?t have solid numbers to show at moment though. Will update 
you on that.

For now, the main advantage of having DPDK version of Vmxnet3 driver is having 
scalability across multiple hypervisors, and application portability keeping in 
mind that backend can be optimized and higher throughput can be achieved at 
later stage.

Thanks,
Rashmin

From: hyunseok.chang at gmail.com [mailto:hyunseok.ch...@gmail.com] On Behalf 
Of Hyunseok
Sent: Monday, July 07, 2014 4:49 PM
To: Patel, Rashmin N; dev at dpdk.org
Subject: RE: [dpdk-dev] Performance issue with vmxnet3 pmd


Thanks for your response.

I am actually more interested in stock (non-dpdk) vmxnet3 driver vs. vmxnet3 
pmd driver comparison.

When I forward pkts from stock vmxnet3 driver, I am able to achieve much higher 
throughput than with vmxnet3 pmd.  To make comparison fair, I did not leverage 
gro/gso.

Does any of the overheads you mentioned play a role in this comparison?  Here I 
am comparing different drivers for the same vmxnet3 interface...

Regards,
Hyunseok
On Jul 7, 2014 7:03 PM, "Patel, Rashmin N" mailto:rashmin.n.patel at intel.com>> wrote:
Hi Hyunseok,

We should not compare Vmxnet3-PMD with ixgbe-PMD performance as Vmxnet3 device 
is a para-virtual device and it's not similar to directly assigned device to a 
VM either.
There is VMEXIT/VMEXIT occurrence at burst-size boundary and that overhead 
can?t be eliminated unless the design of Vmxnet3 is updated in future. In 
addition to that the packets is being touched in ESXi hypervisor vSwitch layer 
between physical NIC and a virtual machine, which introduces extra overhead, 
which you won't have in case of using Niantic being used natively or passed 
through Vt-d to a virtual machine.

Feature wise, we can compare it to Virtio-PMD solution, but again there is a 
little different in device handling and backend driver support compared to 
Vmxnet3 device so performance comparison won?t to apple to apple.

Thanks,
Rashmin

-Original Message-
From: dev [mailto:dev-bounces at dpdk.org<mailto:dev-boun...@dpdk.org>] On 
Behalf Of Hyunseok
Sent: Monday, July 07, 2014 3:22 PM
To: dev at dpdk.org<mailto:dev at dpdk.org>
Subject: [dpdk-dev] Performance issue with vmxnet3 pmd

Hi,

I was testing l2-fwd with vmxnet3 pmd (included in dpdk).

The maximum forwarding rate I got from vmxnet3 pmd with l2fwd is only 2.5 to 
2.8 Gbps.

This is in contrast with ixgbe pmd with which I could easily achieve 10 gbps 
forwarding rate.

With the original vmxnet3 driver (non pmd), I could also achieve close to
10 gpbs with multiple iperf.   But I can never achieve that rate with
vmxnet3 pmd...

So basically vmxnet3 pmd doesn't seem that fast.  Is this a known issue?

Thanks,
-Hyunseok

[dpdk-dev] Performance issue with vmxnet3 pmd

[Patel, Rashmin N]

Hi Hyunseok,

We should not compare Vmxnet3-PMD with ixgbe-PMD performance as Vmxnet3 device 
is a para-virtual device and it's not similar to directly assigned device to a 
VM either. 
There is VMEXIT/VMEXIT occurrence at burst-size boundary and that overhead 
can?t be eliminated unless the design of Vmxnet3 is updated in future. In 
addition to that the packets is being touched in ESXi hypervisor vSwitch layer 
between physical NIC and a virtual machine, which introduces extra overhead, 
which you won't have in case of using Niantic being used natively or passed 
through Vt-d to a virtual machine.

Feature wise, we can compare it to Virtio-PMD solution, but again there is a 
little different in device handling and backend driver support compared to 
Vmxnet3 device so performance comparison won?t to apple to apple.

Thanks,
Rashmin

-Original Message-
From: dev [mailto:dev-boun...@dpdk.org] On Behalf Of Hyunseok
Sent: Monday, July 07, 2014 3:22 PM
To: dev at dpdk.org
Subject: [dpdk-dev] Performance issue with vmxnet3 pmd

Hi,

I was testing l2-fwd with vmxnet3 pmd (included in dpdk).

The maximum forwarding rate I got from vmxnet3 pmd with l2fwd is only 2.5 to 
2.8 Gbps.

This is in contrast with ixgbe pmd with which I could easily achieve 10 gbps 
forwarding rate.

With the original vmxnet3 driver (non pmd), I could also achieve close to
10 gpbs with multiple iperf.   But I can never achieve that rate with
vmxnet3 pmd...

So basically vmxnet3 pmd doesn't seem that fast.  Is this a known issue?

Thanks,
-Hyunseok

[dpdk-dev] multiple VLAN IDs for SR-IOV ports

[Patel, Rashmin N]

I haven't checked it with ip tool. But I just verified from the Niantic specs, 
and I see PFVLVF[64] and PFVLVFB[128] - 32 bit register arrays to configure 
VLAN-Pool/VF mapping.
PFVLVF[i] holds a vlan_id need to be mapped
PFVLVFB[2i] and PFVLVFB[2i+1] holds bitmap for max possible pools (64 bits) to 
map any of those pools to the PFVLVF[i].vlan_id

So technically you should be able to map it, the only thing I'm not sure if 
it's supported with ip tool or not.
If you quickly want to check it out, you can just write those registers through 
PCI bar.

Thanks,
Rashmin

-Original Message-
From: dev [mailto:dev-boun...@dpdk.org] On Behalf Of James Yu
Sent: Sunday, February 02, 2014 11:53 PM
To: Jose Gavine Cueto
Cc: dev at dpdk.org
Subject: Re: [dpdk-dev] multiple VLAN IDs for SR-IOV ports

Patel,

I can use the following command to add one VLAN with VID of 3 to VF 0 of
eth6

ip link set eth6 vf 0 vlan 3

Could you show me an example of adding 2 VLAN IDs such as VID of 3 and 13 to 
the same VF 0 ? I want VF 0 to pass traffic from VLAN 3 and 13. How do I do 
that ?
>From the man page of ip, I do not see it describes about adding more than one 
>VLAN IDs to the same VF port.
Can I use this ?
ip link set eth6 vf0 vlan 3,13


Thanks

James


On Sat, Feb 1, 2014 at 9:13 AM, Jose Gavine Cueto wrote:

> hi,
>
>
> On Thu, Jan 30, 2014 at 2:45 PM, James Yu  wrote:
>
>> Any one know what commands to use on the KVM host to add multiple 
>> VLAN IDs to a SR-IOV port ?
>>
>> I could only add one VLAN ID using
>>
>> ip link set ethN vf NUM vlan VLANID
>>
>
> What do you mean by "multiple VLAN IDs" ?  ethN and vf NUM here let's 
> you add more than one vlan id.
>
>
>>
>> Also do I have to call any routine on the DPDK side to add those VIDs ?
>>
>> James
>>
>
>
>
> --
> To stop learning is like to stop loving.
>

[dpdk-dev] L2fwd Performance issue with Virtual Machine

[Patel, Rashmin N]

Vincent, I agree with your explanation that when you move to PMD from regular 
interrupt based driver, you would definitely get better performance of a Linux 
guest until you reach the bottleneck of the vSwitch. But I said "optimal 
performance benefit having PMD" which is only possible if vmxnet3 backend 
driver has support for vmxnet3-PMD frontend driver inside guest and you never 
know if vmware helps adding support for the same or not but there is a way 
OPEN. 

The motive is of having PMD of para-virtual devices is to get performance close 
to shared memory solution while supporting standard devices, I believe and 
being at Intel, we strive for the optimal solution and pardon me if I created 
any confusion.

Thanks,
Rashmin
-Original Message-
From: dev [mailto:dev-boun...@dpdk.org] On Behalf Of Vincent JARDIN
Sent: Friday, October 04, 2013 11:33 PM
To: dev at dpdk.org
Subject: Re: [dpdk-dev] L2fwd Performance issue with Virtual Machine

I disagree Rashmin. We did measurements with 64 bytes packets: the Linux kernel 
of the guest is the bottleneck, so the vmxnet3 PMD helps to increase the packet 
rate of the Linux guests.

PMD helps within guest for packet rate until you rich (of course) the 
bottleneck of the host's vSwitch.

In order to accelerate the host's vSwitch, you have to run a fast path based 
vSwitch on the host too.

Best regards,
   Vincent

On 04/10/2013 23:36, Selvaganapathy Chidambaram wrote:
> Thanks Rashmin for your time and help!
>
> So it looks like with the given hardware config, we could probably 
> only achieve around 8 Gbps in VM without using SRIOV. Once DPDK is 
> used in vSwitch design, we could gain more performance.
>
>
> Thanks,
> Selvaganapathy.C.
>
>
> On Fri, Oct 4, 2013 at 11:02 AM, Patel, Rashmin N 
> > wrote:
>
>> Correction: "you would NOT get optimal performance benefit having PMD"
>>
>> Thanks,
>> Rashmin
>> -Original Message-
>> From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Patel, Rashmin N
>> Sent: Friday, October 04, 2013 10:47 AM
>> To: Selvaganapathy Chidambaram
>> Cc: dev at dpdk.org
>> Subject: Re: [dpdk-dev] L2fwd Performance issue with Virtual Machine
>>
>> Hi,
>>
>> If you are not using SRIOV or direct device assignment to VM, your 
>> traffic hits vSwitch(via vmware native ixgbe driver and network 
>> stack) in the ESX and switched to your E1000/VMXNET3 interface 
>> connected to a VM. The vSwitch is not optimized for PMD at present so 
>> you would get optimal performance benefit having PMD, I believe.
>>
>> For the RSS front, I would say you won't see much difference with RSS 
>> enabled for 1500 bytes frames. In fact, core is capable of handling 
>> such traffic in VM, but the bottleneck is in ESXi software switching 
>> layer, that's what my initial research shows across multiple hypervisors.
>>
>> Thanks,
>> Rashmin
>>
>> -Original Message-
>> From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of Selvaganapathy 
>> Chidambaram
>> Sent: Thursday, October 03, 2013 2:39 PM
>> To: dev at dpdk.org
>> Subject: [dpdk-dev] L2fwd Performance issue with Virtual Machine
>>
>> Hello Everyone,
>>
>> I have tried to run DPDK sample application l2fwd(modified to support 
>> multiple queues) in my ESX Virtual Machine. I see that performance is 
>> not scaling with cores. [My apologies for the long email]
>>
>> *Setup:*
>>
>> Connected VM to two ports of Spirent with 10Gig link. Sent 10 Gig 
>> traffic of L3 packet of length 1500 bytes (with four different flows) 
>> from Spirent through one port and received at the second port. Also 
>> sent traffic from reverse direction so that net traffic is 20 Gbps. Haven't 
>> enabled SR-IOV or
>>   Direct path I/O.
>>
>> *Emulated Driver:*
>>
>> With default emulated driver, I got 7.3 Gbps for 1 core. Adding 
>> multiple cores did not improve the performance. On debugging I 
>> noticed that function
>> eth_em_infos_get() says RSS is not supported.
>>
>> *vmxnet3_usermap:*
>>
>> Then I tried extension vmxnet3_usermap and got 8.7 Gbps for 1 core. 
>> Again adding another core did not help. On debugging, I noticed that 
>> in vmxnet3 kernel driver (in function vmxnet3_probe_device) , RSS is 
>> disabled if *
>> adapter->is_shm* is non zero. In our case, its 
>> adapter->VMXNET3_SHM_USERMAP_DRIVER
>> which is non zero.
>>
>> Before trying to enable it, I would like to know if there is any 
>> known limitation why RSS is not enabled in both the drivers. Please 
>> help me understand.
>>
>> *Hardware Configuration:*
>> Hardware  : Intel Xeon 2.4 Ghz 4 CPUs
>> Hyperthreading  : No
>> RAM : 16 GB
>> Hypervisor : ESXi 5.1
>> Ethernet: Intel 82599EB 10 Gig SFP
>>
>>
>> Guest VM : 2 vCPU, 2 GB RAM
>> GuestOS  : Centos 6.2 32 bit
>>
>> Thanks in advance for your time and help!!!
>>
>> Thanks,
>> Selva.
>>

[dpdk-dev] L2fwd Performance issue with Virtual Machine

[Patel, Rashmin N]

Correction: "you would NOT get optimal performance benefit having PMD"

Thanks,
Rashmin
-Original Message-
From: dev [mailto:dev-boun...@dpdk.org] On Behalf Of Patel, Rashmin N
Sent: Friday, October 04, 2013 10:47 AM
To: Selvaganapathy Chidambaram
Cc: dev at dpdk.org
Subject: Re: [dpdk-dev] L2fwd Performance issue with Virtual Machine

Hi,

If you are not using SRIOV or direct device assignment to VM, your traffic hits 
vSwitch(via vmware native ixgbe driver and network stack) in the ESX and 
switched to your E1000/VMXNET3 interface connected to a VM. The vSwitch is not 
optimized for PMD at present so you would get optimal performance benefit 
having PMD, I believe.

For the RSS front, I would say you won't see much difference with RSS enabled 
for 1500 bytes frames. In fact, core is capable of handling such traffic in VM, 
but the bottleneck is in ESXi software switching layer, that's what my initial 
research shows across multiple hypervisors.

Thanks,
Rashmin

-Original Message-
From: dev [mailto:dev-boun...@dpdk.org] On Behalf Of Selvaganapathy Chidambaram
Sent: Thursday, October 03, 2013 2:39 PM
To: dev at dpdk.org
Subject: [dpdk-dev] L2fwd Performance issue with Virtual Machine

Hello Everyone,

I have tried to run DPDK sample application l2fwd(modified to support multiple 
queues) in my ESX Virtual Machine. I see that performance is not scaling with 
cores. [My apologies for the long email]

*Setup:*

Connected VM to two ports of Spirent with 10Gig link. Sent 10 Gig traffic of L3 
packet of length 1500 bytes (with four different flows) from Spirent through 
one port and received at the second port. Also sent traffic from reverse 
direction so that net traffic is 20 Gbps. Haven't enabled SR-IOV or  Direct 
path I/O.

*Emulated Driver:*

With default emulated driver, I got 7.3 Gbps for 1 core. Adding multiple cores 
did not improve the performance. On debugging I noticed that function 
eth_em_infos_get() says RSS is not supported.

*vmxnet3_usermap:*

Then I tried extension vmxnet3_usermap and got 8.7 Gbps for 1 core. Again 
adding another core did not help. On debugging, I noticed that in vmxnet3 
kernel driver (in function vmxnet3_probe_device) , RSS is disabled if *
adapter->is_shm* is non zero. In our case, its 
adapter->VMXNET3_SHM_USERMAP_DRIVER
which is non zero.

Before trying to enable it, I would like to know if there is any known 
limitation why RSS is not enabled in both the drivers. Please help me 
understand.

*Hardware Configuration:*
Hardware  : Intel Xeon 2.4 Ghz 4 CPUs
Hyperthreading  : No
RAM : 16 GB
Hypervisor : ESXi 5.1
Ethernet: Intel 82599EB 10 Gig SFP


Guest VM : 2 vCPU, 2 GB RAM
GuestOS  : Centos 6.2 32 bit

Thanks in advance for your time and help!!!

Thanks,
Selva.

[dpdk-dev] L2fwd Performance issue with Virtual Machine

[Patel, Rashmin N]

Hi,

If you are not using SRIOV or direct device assignment to VM, your traffic hits 
vSwitch(via vmware native ixgbe driver and network stack) in the ESX and 
switched to your E1000/VMXNET3 interface connected to a VM. The vSwitch is not 
optimized for PMD at present so you would get optimal performance benefit 
having PMD, I believe.

For the RSS front, I would say you won't see much difference with RSS enabled 
for 1500 bytes frames. In fact, core is capable of handling such traffic in VM, 
but the bottleneck is in ESXi software switching layer, that's what my initial 
research shows across multiple hypervisors.

Thanks,
Rashmin

-Original Message-
From: dev [mailto:dev-boun...@dpdk.org] On Behalf Of Selvaganapathy Chidambaram
Sent: Thursday, October 03, 2013 2:39 PM
To: dev at dpdk.org
Subject: [dpdk-dev] L2fwd Performance issue with Virtual Machine

Hello Everyone,

I have tried to run DPDK sample application l2fwd(modified to support multiple 
queues) in my ESX Virtual Machine. I see that performance is not scaling with 
cores. [My apologies for the long email]

*Setup:*

Connected VM to two ports of Spirent with 10Gig link. Sent 10 Gig traffic of L3 
packet of length 1500 bytes (with four different flows) from Spirent through 
one port and received at the second port. Also sent traffic from reverse 
direction so that net traffic is 20 Gbps. Haven't enabled SR-IOV or  Direct 
path I/O.

*Emulated Driver:*

With default emulated driver, I got 7.3 Gbps for 1 core. Adding multiple cores 
did not improve the performance. On debugging I noticed that function 
eth_em_infos_get() says RSS is not supported.

*vmxnet3_usermap:*

Then I tried extension vmxnet3_usermap and got 8.7 Gbps for 1 core. Again 
adding another core did not help. On debugging, I noticed that in vmxnet3 
kernel driver (in function vmxnet3_probe_device) , RSS is disabled if *
adapter->is_shm* is non zero. In our case, its 
adapter->VMXNET3_SHM_USERMAP_DRIVER
which is non zero.

Before trying to enable it, I would like to know if there is any known 
limitation why RSS is not enabled in both the drivers. Please help me 
understand.

*Hardware Configuration:*
Hardware  : Intel Xeon 2.4 Ghz 4 CPUs
Hyperthreading  : No
RAM : 16 GB
Hypervisor : ESXi 5.1
Ethernet: Intel 82599EB 10 Gig SFP


Guest VM : 2 vCPU, 2 GB RAM
GuestOS  : Centos 6.2 32 bit

Thanks in advance for your time and help!!!

Thanks,
Selva.

[dpdk-dev] vmxnet3-usermap related query

[Patel, Rashmin N]

Hi,


I was trying benchmark vmxnet3-usermap solution with the required dpdk release. 
I could not compile the kernel module in kmod directory on 3.6.0 kernel. I 
wonder if anyone has faced this kind of issue. Let me know if I'm

missing something which is not in the documentation. And  does anyone know 
which kernel versions are supported for vmxnet3 module? I looked at the 
documentation but it doesn't provide any kernel version dependency. I would 
like to execute it first to get feeling about performance.



Appreciating your help.



Thanks,

Rashmin



-- next part --
An HTML attachment was scrubbed...
URL: