[dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x?

[Prashant Upadhyaya]

Hi Stephen,

Can you please advise on your experience of the kind of data rates you have 
been able to achieve with vmxnet3.
Also did you have to do any special optimizations at the vmnic of ESXi for the 
above, kindly let me know.

Regards
-Prashant


-Original Message-
From: dev [mailto:dev-boun...@dpdk.org] On Behalf Of Prashant Upadhyaya
Sent: Tuesday, March 11, 2014 10:57 AM
To: Stephen Hemminger
Cc: dev at dpdk.org
Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x?

Hi Stephen,

This is great news !
I can wait for a formal release of DPDK with your driver.
Please let me know when is the release expected. I will happily migrate to that.

Regards
-Prashant


-Original Message-
From: Stephen Hemminger [mailto:step...@networkplumber.org]
Sent: Monday, March 10, 2014 9:21 PM
To: Prashant Upadhyaya
Cc: Srinivasan J; dev at dpdk.org
Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x?

On Mon, 10 Mar 2014 13:30:48 +0530
Prashant Upadhyaya  wrote:

> Hi Srini,
>
> Thanks, I could also make it work, thanks to your cue !
>
> Now then, this multi-segment not being supported in vmxnet3 driver is a big 
> party-pooper for me. Unfortunately in my usecase, I do indeed make heavy use 
> of multisegment buffers for sending out the data, so my usecase has failed 
> and I will have to fix that.
>
> Also, can you please adivse how much is the max data rates you have been able 
> to achieve with one vmxnet3 10G port.
>
> Thanks a lot for the advice once again.
>
> Regards
> -Prashant

I am integrating our driver with the 1.6.1 DPDK driver.
We support multi-segment, if you want I will backport that feature first.





===
Please refer to http://www.aricent.com/legal/email_disclaimer.html
for important disclosures regarding this electronic communication.
===




===
Please refer to http://www.aricent.com/legal/email_disclaimer.html
for important disclosures regarding this electronic communication.
===

[dpdk-dev] RSS, performance, and stability issues vmxnet3-pmd

[Daniel Kan]

I?m unable to get RSS to work properly with vmxnet3-pmd. The first issue is 
that the number of rxqs must be power of 2. Otherwise, rte_eth_dev_start() 
fails due to inability to activate vmxnet3 NIC. This is not too big of a deal, 
but physical NICs don?t have this requirement. 

The second issue is that RSS is just not working at all for me. The rxmode is 
set to ETH_MQ_RX_RSS and rss_hf = ETH_RSS_IPV4_TCP | ETH_RSS_IPV4_UDP | 
ETH_RSS_IPV4 | ETH_RSS_IPV6. The same configuration works for a real NIC. When 
I checked mb->pkt.hash, the value is all zeroed out. 

Even if I disabled RSS, I found the performance of vmxnet3-pmd to be quite 
poor, peaking out at 600k pps with 64 byte packet, while libpcap can do 650k 
pps. 

Lastly, there is a stability issue. On a number of occasions, vmxnet3-pmd stops 
receiving packets after some random time and several million packets. 

I?m not sure if anyone else is having as much issue as I?m, I will give 
vmxnet3-usermap a try. 

Finally, does either vmxnet3-usermap or vmxnet3-pmd work well for 
non-Intel-based underlying physical NIC? 

Thanks. 

Dan

[dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x?

[Prashant Upadhyaya]

Hi Stephen,

This is great news !
I can wait for a formal release of DPDK with your driver.
Please let me know when is the release expected. I will happily migrate to that.

Regards
-Prashant


-Original Message-
From: Stephen Hemminger [mailto:step...@networkplumber.org]
Sent: Monday, March 10, 2014 9:21 PM
To: Prashant Upadhyaya
Cc: Srinivasan J; dev at dpdk.org
Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x?

On Mon, 10 Mar 2014 13:30:48 +0530
Prashant Upadhyaya  wrote:

> Hi Srini,
>
> Thanks, I could also make it work, thanks to your cue !
>
> Now then, this multi-segment not being supported in vmxnet3 driver is a big 
> party-pooper for me. Unfortunately in my usecase, I do indeed make heavy use 
> of multisegment buffers for sending out the data, so my usecase has failed 
> and I will have to fix that.
>
> Also, can you please adivse how much is the max data rates you have been able 
> to achieve with one vmxnet3 10G port.
>
> Thanks a lot for the advice once again.
>
> Regards
> -Prashant

I am integrating our driver with the 1.6.1 DPDK driver.
We support multi-segment, if you want I will backport that feature first.





===
Please refer to http://www.aricent.com/legal/email_disclaimer.html
for important disclosures regarding this electronic communication.
===

[dpdk-dev] Setting up hugepage memory failed when there is a java process also using hugepages

[Carlos Franco]

Hello
I am having some rare problem with hugepages. I am running DPDK 1.5.1 in CenOS 
6.4. Everything is running smothly and we have done quite good progress 
integrating DPDK into our solution. But sometimes, there is an error with the 
init of the huge pages. The problem is when there is another process started 
that also used hugepages, in this case, a java 1.7 process (jdk 1.7.0_45 
64-bits).
The machine only has one socket with 1 Xeon CPU with 4 cores and 4 GB. I have 
configured 256 pages of 2MB. Everything is running in 64 bits. 
grep Huge /proc/meminfo
AnonHugePages:253952 kB
HugePages_Total: 256
HugePages_Free:  256
HugePages_Rsvd:0
HugePages_Surp:0
Hugepagesize:   2048 kB
When I start the DPDK process (with ?c 01 ?n 4 ?m 32 options), this is the 
trace:
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Setting up hugepage memory...
EAL: Ask a virtual area of 0x394264576 bytes
EAL: Virtual area found at 0x7fc0c900 (size = 0x1780)
EAL: Ask a virtual area of 0x4194304 bytes
EAL: Virtual area found at 0x7fc0c8a0 (size = 0x40)
EAL: Ask a virtual area of 0x132120576 bytes
EAL: Virtual area found at 0x7fc0c0a0 (size = 0x7e0)
EAL: Ask a virtual area of 0x4194304 bytes
EAL: Virtual area found at 0x7fc0c040 (size = 0x40)
EAL: Ask a virtual area of 0x2097152 bytes
EAL: Virtual area found at 0x7fc0c000 (size = 0x20)
EAL: Requesting 16 pages of size 2MB from socket 0
...
Everything goes well. After the start, this is the hugepages information:
AnonHugePages:280576 kB
HugePages_Total: 256
HugePages_Free:  220
HugePages_Rsvd:   35
HugePages_Surp:0
Hugepagesize:   2048 kB

If I started then the java process, it starts ok and this is the hugepages 
information:

HugePages_Total: 256
HugePages_Free:  204
HugePages_Rsvd:   74
HugePages_Surp:0
Hugepagesize:   2048 kB

If I stop and start the DPDK process, it fails with the following trace:

EAL: Detected lcore 0 as core 0 on socket 0
EAL: Setting up hugepage memory...
EAL: map_all_hugepages(): mmap failed: Cannot allocate memory
EAL: Failed to mmap 2 MB hugepages
PANIC in rte_eal_init():
Cannot init memory

And after the fail, this is the hugepage information:

AnonHugePages:251904 kB
HugePages_Total: 256
HugePages_Free:   40
HugePages_Rsvd:   40
HugePages_Surp:0
Hugepagesize:   2048 kB

It seems that it tries to map all the hugepages even if it should be limited 
with ?m 32. After the fail, there are 201 rtemap_X files in /mnt/huge.

When I stop the java process, the DPDK process starts without any problem. I 
don?t know if this happens also with any other process different from java and 
that uses hugepages.

Am I doing something wrong? 

Thanks a lot

Carlos

[dpdk-dev] [memnic PATCH 5/5] pmd: handle multiple segments on xmit

[Hiroshi Shimamoto]

From: Hiroshi Shimamoto 

The current MEMNIC PMD cannot handle multiple segments.

Add the functionality to transmit a mbuf which has multiple segments.
Walk every segment in transmitting mbuf and copy the data to MEMNIC
packet buffer.

Signed-off-by: Hiroshi Shimamoto 
Reviewed-by: Hayato Momma 
---
 pmd/pmd_memnic.c | 19 ++-
 1 file changed, 14 insertions(+), 5 deletions(-)

diff --git a/pmd/pmd_memnic.c b/pmd/pmd_memnic.c
index abfd437..4ee655d 100644
--- a/pmd/pmd_memnic.c
+++ b/pmd/pmd_memnic.c
@@ -324,9 +324,11 @@ static uint16_t memnic_xmit_pkts(void *tx_queue,
pkts = bytes = errs = 0;

for (nr = 0; nr < nb_pkts; nr++) {
-   int len = rte_pktmbuf_data_len(tx_pkts[nr]);
+   int pkt_len = rte_pktmbuf_pkt_len(tx_pkts[nr]);
+   struct rte_mbuf *sg;
+   void *ptr;

-   if (len > MEMNIC_MAX_FRAME_LEN) {
+   if (pkt_len > MEMNIC_MAX_FRAME_LEN) {
errs++;
break;
}
@@ -356,12 +358,19 @@ retry:
idx = 0;
adapter->down_idx = idx;

-   p->len = len;
+   p->len = pkt_len;

-   rte_memcpy(p->data, rte_pktmbuf_mtod(tx_pkts[nr], void *), len);
+   ptr = p->data;
+   for (sg = tx_pkts[nr]; sg; sg = sg->pkt.next) {
+   void *src = rte_pktmbuf_mtod(sg, void *);
+   int data_len = sg->pkt.data_len;
+
+   rte_memcpy(ptr, src, data_len);
+   ptr += data_len;
+   }

pkts++;
-   bytes += len;
+   bytes += pkt_len;

rte_mb();
p->status = MEMNIC_PKT_ST_FILLED;
-- 
1.8.4

[dpdk-dev] [memnic PATCH 4/5] pmd: account statistics

[Hiroshi Shimamoto]

From: Hiroshi Shimamoto 

Implement packet accounting of MEMNIC on TX/RX.

Signed-off-by: Hiroshi Shimamoto 
Reviewed-by: Hayato Momma 
---
 pmd/pmd_memnic.c | 37 +++--
 1 file changed, 35 insertions(+), 2 deletions(-)

diff --git a/pmd/pmd_memnic.c b/pmd/pmd_memnic.c
index fc2d990..abfd437 100644
--- a/pmd/pmd_memnic.c
+++ b/pmd/pmd_memnic.c
@@ -255,18 +255,23 @@ static uint16_t memnic_recv_pkts(void *rx_queue,
struct memnic_packet *p;
struct rte_mbuf *mb;
uint16_t nr;
+   uint64_t pkts, bytes, errs;
int idx;
+   struct rte_eth_stats *st = >stats[rte_lcore_id()];

if (!adapter->nic->hdr.valid)
return 0;

+   pkts = bytes = errs = 0;
idx = adapter->up_idx;
for (nr = 0; nr < nb_pkts; nr++) {
p = >packets[idx];
if (p->status != MEMNIC_PKT_ST_FILLED)
break;
-   if (p->len > MEMNIC_MAX_FRAME_LEN)
+   if (p->len > MEMNIC_MAX_FRAME_LEN) {
+   errs++;
goto drop;
+   }
mb = rte_pktmbuf_alloc(adapter->mp);
if (!mb)
break;
@@ -279,6 +284,9 @@ static uint16_t memnic_recv_pkts(void *rx_queue,
mb->pkt.data_len = p->len;
rx_pkts[nr] = mb;

+   pkts++;
+   bytes += p->len;
+
 drop:
rte_mb();
p->status = MEMNIC_PKT_ST_FREE;
@@ -288,6 +296,13 @@ drop:
}
adapter->up_idx = idx;

+   /* stats */
+   st->ipackets += pkts;
+   st->ibytes += bytes;
+   st->ierrors += errs;
+   st->q_ipackets[0] += pkts;
+   st->q_ibytes[0] += bytes;
+
return nr;
 }

@@ -300,14 +315,21 @@ static uint16_t memnic_xmit_pkts(void *tx_queue,
struct memnic_packet *p;
uint16_t nr;
int idx, old;
+   struct rte_eth_stats *st = >stats[rte_lcore_id()];
+   uint64_t pkts, bytes, errs;

if (!adapter->nic->hdr.valid)
return 0;

+   pkts = bytes = errs = 0;
+
for (nr = 0; nr < nb_pkts; nr++) {
int len = rte_pktmbuf_data_len(tx_pkts[nr]);
-   if (len > MEMNIC_MAX_FRAME_LEN)
+
+   if (len > MEMNIC_MAX_FRAME_LEN) {
+   errs++;
break;
+   }
 retry:
idx = ACCESS_ONCE(adapter->down_idx);
p = >packets[idx];
@@ -315,6 +337,7 @@ retry:
if (old != MEMNIC_PKT_ST_FREE) {
if (old == MEMNIC_PKT_ST_FILLED &&
idx == ACCESS_ONCE(adapter->down_idx)) {
+   errs++;
break;
}
goto retry;
@@ -337,12 +360,22 @@ retry:

rte_memcpy(p->data, rte_pktmbuf_mtod(tx_pkts[nr], void *), len);

+   pkts++;
+   bytes += len;
+
rte_mb();
p->status = MEMNIC_PKT_ST_FILLED;

rte_pktmbuf_free(tx_pkts[nr]);
}

+   /* stats */
+   st->opackets += pkts;
+   st->obytes += bytes;
+   st->oerrors += errs;
+   st->q_opackets[0] += pkts;
+   st->q_obytes[0] += bytes;
+
return nr;
 }

-- 
1.8.4

[dpdk-dev] [memnic PATCH 3/5] pmd: implement stats of MEMNIC

[Hiroshi Shimamoto]

From: Hiroshi Shimamoto 

Implement missing feature to account statistics.
This patch adds just an infrastructure.

Signed-off-by: Hiroshi Shimamoto 
Reviewed-by: Hayato Momma 
---
 pmd/pmd_memnic.c | 45 ++---
 1 file changed, 42 insertions(+), 3 deletions(-)

diff --git a/pmd/pmd_memnic.c b/pmd/pmd_memnic.c
index bf5fc2e..fc2d990 100644
--- a/pmd/pmd_memnic.c
+++ b/pmd/pmd_memnic.c
@@ -51,6 +51,7 @@ struct memnic_adapter {
int up_idx, down_idx;
struct rte_mempool *mp;
struct ether_addr mac_addr;
+   struct rte_eth_stats stats[RTE_MAX_LCORE];
 };

 static inline struct memnic_adapter *get_adapter(const struct rte_eth_dev *dev)
@@ -126,13 +127,51 @@ static void memnic_dev_infos_get(struct rte_eth_dev *dev,
dev_info->max_mac_addrs = 1;
 }

-static void memnic_dev_stats_get(__rte_unused struct rte_eth_dev *dev,
-__rte_unused struct rte_eth_stats *stats)
+static void memnic_dev_stats_get(struct rte_eth_dev *dev,
+struct rte_eth_stats *stats)
 {
+   struct memnic_adapter *adapter = get_adapter(dev);
+   int i;
+
+   memset(stats, 0, sizeof(*stats));
+   for (i = 0; i < RTE_MAX_LCORE; i++) {
+   struct rte_eth_stats *st = >stats[i];
+
+   stats->ipackets += st->ipackets;
+   stats->opackets += st->opackets;
+   stats->ibytes += st->ibytes;
+   stats->obytes += st->obytes;
+   stats->ierrors += st->ierrors;
+   stats->oerrors += st->oerrors;
+   stats->imcasts += st->imcasts;
+   stats->rx_nombuf += st->rx_nombuf;
+   stats->fdirmatch += st->fdirmatch;
+   stats->fdirmiss += st->fdirmiss;
+
+   /* no multiqueue support now */
+   stats->q_ipackets[0] = st->q_ipackets[0];
+   stats->q_opackets[0] = st->q_opackets[0];
+   stats->q_ibytes[0] = st->q_ibytes[0];
+   stats->q_obytes[0] = st->q_obytes[0];
+   stats->q_errors[0] = st->q_errors[0];
+
+   stats->ilbpackets += st->ilbpackets;
+   stats->olbpackets += st->olbpackets;
+   stats->ilbbytes += st->ilbbytes;
+   stats->olbbytes += st->olbbytes;
+   }
 }

-static void memnic_dev_stats_reset(__rte_unused struct rte_eth_dev *dev)
+static void memnic_dev_stats_reset(struct rte_eth_dev *dev)
 {
+   struct memnic_adapter *adapter = get_adapter(dev);
+   int i;
+
+   for (i = 0; i < RTE_MAX_LCORE; i++) {
+   struct rte_eth_stats *st = >stats[i];
+
+   memset(st, 0, sizeof(*st));
+   }
 }

 static int memnic_dev_link_update(struct rte_eth_dev *dev,
-- 
1.8.4

[dpdk-dev] [memnic PATCH 2/5] pmd: check frame length from host

[Hiroshi Shimamoto]

From: Hiroshi Shimamoto 

Drop packets which have invalid length.

Normally this must not happen while vSwitch works fine, however
it's better to put a sentinel to prevent memory corruption.

Signed-off-by: Hiroshi Shimamoto 
Reviewed-by: Hayato Momma 
---
 pmd/pmd_memnic.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/pmd/pmd_memnic.c b/pmd/pmd_memnic.c
index 805f0b2..bf5fc2e 100644
--- a/pmd/pmd_memnic.c
+++ b/pmd/pmd_memnic.c
@@ -226,6 +226,8 @@ static uint16_t memnic_recv_pkts(void *rx_queue,
p = >packets[idx];
if (p->status != MEMNIC_PKT_ST_FILLED)
break;
+   if (p->len > MEMNIC_MAX_FRAME_LEN)
+   goto drop;
mb = rte_pktmbuf_alloc(adapter->mp);
if (!mb)
break;
@@ -238,6 +240,7 @@ static uint16_t memnic_recv_pkts(void *rx_queue,
mb->pkt.data_len = p->len;
rx_pkts[nr] = mb;

+drop:
rte_mb();
p->status = MEMNIC_PKT_ST_FREE;

-- 
1.8.4

[dpdk-dev] [memnic PATCH 1/5] pmd: fix race condition

[Hiroshi Shimamoto]

From: Hiroshi Shimamoto 

There is a race condition, on transmit to vSwitch.

Guest PMD Host
Thread-A Thread-B vSwitch
   |idx=0   |idx=0   |p[0] st!=2
   |cmpxchg ||
   |p[0] st->1  ||
   |idx=1   ||
   |fill data   ||
   |p[0] st->2  ||p[0] st==2
   |||receive data
   |||p[0] st->0
   ||cmpxchg |
   ||success |p[1] st!=2
   ||p[0] st->1  |
  This is BAD

That causes traffic stop.

We have to take care about that race condition with checking
whether current index is correct.

Signed-off-by: Hiroshi Shimamoto 
Reviewed-by: Hayato Momma 
---
 pmd/pmd_memnic.c | 9 +
 1 file changed, 9 insertions(+)

diff --git a/pmd/pmd_memnic.c b/pmd/pmd_memnic.c
index 30d5a1b..805f0b2 100644
--- a/pmd/pmd_memnic.c
+++ b/pmd/pmd_memnic.c
@@ -278,6 +278,15 @@ retry:
goto retry;
}

+   if (idx != ACCESS_ONCE(adapter->down_idx)) {
+   /*
+* vSwitch freed this and got false positive,
+* need to recover the status and retry.
+*/
+   p->status = MEMNIC_PKT_ST_FREE;
+   goto retry;
+   }
+
if (++idx >= MEMNIC_NR_PACKET)
idx = 0;
adapter->down_idx = idx;
-- 
1.8.4

[dpdk-dev] On vmxnet-pmd crash in DPDK 1.6.0r1

[Daniel Kan]

virtio-pmd has the same pattern. I wonder if vmxnet3-pmd just blindly copied 
the same paradigm. 

lib/librte_pmd_virtio/virtio_ethdev.c
473 static struct eth_driver rte_virtio_pmd = {
474 {
475 .name = "rte_virtio_pmd",
476 .id_table = pci_id_virtio_map,
477 #ifdef RTE_EAL_UNBIND_PORTS
478 .drv_flags = RTE_PCI_DRV_NEED_IGB_UIO,
479 #endif
480 },
481 .eth_dev_init = eth_virtio_dev_init,
482 .dev_private_size = sizeof(struct virtio_adapter),
483 };


On Mar 10, 2014, at 11:20 PM, Daniel Kan  wrote:

> Upon further trace, I know what caused it. The uio hardware resources were 
> never memory mapped when RTE_EAL_UNBIND_PORTS is not enabled. Specifically, 
> pci_dev->mem_resource[] is not mapped.  This explains why setting 
> CONFIG_RTE_EAL_UNBIND_PORTS=y fixes the problem. 
>  
> lib/librte_pmd_vmxnet3/vmxnet3_ethdev.c
> 266 static struct eth_driver rte_vmxnet3_pmd = {
> 267 {
> 268 .name = "rte_vmxnet3_pmd",
> 269 .id_table = pci_id_vmxnet3_map,
> 270 #ifdef RTE_EAL_UNBIND_PORTS
> 271 .drv_flags = RTE_PCI_DRV_NEED_IGB_UIO,
> 272 #endif
> 273 },
> 274 .eth_dev_init = eth_vmxnet3_dev_init,
> 275 .dev_private_size = sizeof(struct vmxnet3_adapter),
> 276 };
> 
> Note drv_flags will be 0. 
> 
> lib/librte_eal/linuxapp/eal/eal_pci.c
> 1039 #ifdef RTE_EAL_UNBIND_PORTS
> 1040 if (dr->drv_flags & RTE_PCI_DRV_NEED_IGB_UIO) {
> 1041 /* unbind driver and load uio resources for Intel NICs */
> 1042 if (pci_switch_module(dr, dev, 1, IGB_UIO_NAME) < 0)
> ...
> 1050 #else
> 1051 if (dr->drv_flags & RTE_PCI_DRV_NEED_IGB_UIO)
> 1052 /* just map resources for Intel NICs */
> 1053 if (pci_uio_map_resource(dev) < 0)
> 1054 return -1;
> 1055 #endif
> 
> If RTE_EAL_UNBIND_PORTS  is defined, pci_switch_module will call 
> pci_uio_map_resource.
> 
> I then looked at the bsdapp?s version, it simply has:
> lib/librte_eal/bsdapp/eal/eal_pci.c
> 479 /* just map the NIC resources */
> 480 if (pci_uio_map_resource(dev) < 0)
> 481 return -1;
> 
> I don?t know the history behind why .drv_flags = RTE_PCI_DRV_NEED_IGB_UIO 
> only for RTE_EAL_UNBIND_PORTS. Can we just ensure pci_uio_map_resource is 
> called in linux just like bsd version? One way is to ensure drv_flags is 
> always set to RTE_PCI_DRV_NEED_IGB_UIO, but I don?t know if this fix will 
> break other parts. 
> 
> +++ vmxnet3_ethdev.c  2014-03-10 23:18:02.087742434 -0700
> @@ -267,9 +267,7 @@
>   {
>   .name = "rte_vmxnet3_pmd",
>   .id_table = pci_id_vmxnet3_map,
> -#ifdef RTE_EAL_UNBIND_PORTS
>   .drv_flags = RTE_PCI_DRV_NEED_IGB_UIO,
> -#endif
>   },
>   .eth_dev_init = eth_vmxnet3_dev_init,
>   .dev_private_size = sizeof(struct vmxnet3_adapter),
> 
> Thanks. 
> 
> Dan
> 
> On Mar 10, 2014, at 10:33 PM, Daniel Kan  wrote:
> 
>> I also got the segfault. Setting CONFIG_RTE_EAL_UNBIND_PORTS=y is not ideal 
>> because it would also unbind my VMNET3 management interface.
>> 
>> Does anyone know why the crash is happening? According to the stacktrace, 
>> hw_addrX is 0x0 during eth_vmxnet3_dev_init. 
>> 
>> I?m running on ubuntu 12.04LTS on ESXi 5.5 with Intel I350 NIC as the 
>> physical adapter for VMNET3. 
>> 
>> Program received signal SIGSEGV, Segmentation fault.
>> 0x004fe0eb in vmxnet3_read_addr (addr=0x0) at 
>> /home/dkan/nyansa/3rd-party/dpdk-1.6.0r1/lib/librte_pmd_vmxnet3/vmxnet3_ethdev.h:139
>> (gdb) up
>> #1  0x004fe331 in eth_vmxnet3_dev_init (eth_drv=0x799440 
>> , eth_dev=0x7d5280 ) at 
>> /home/dkan/nyansa/3rd-party/dpdk-1.6.0r1/lib/librte_pmd_v\
>> mxnet3/vmxnet3_ethdev.c:218
>> (gdb) print *hw
>> $12 = {hw_addr0 = 0x0, hw_addr1 = 0x0, back = 0x0, device_id = 1968, 
>> vendor_id = 5549, subsystem_device_id = 0, subsystem_vendor_id = 0, 
>> adapter_stopped = 0, perm_addr = "\000\000\\
>> 000\000\000", num_tx_queues = 1 '\001', num_rx_queues = 1 '\001', 
>> bufs_per_pkt = 1 '\001', cur_mtu = 1500, tqd_start = 0x0, rqd_start = 0x0, 
>> shared = 0x0, sharedPA = 0, queueDescPA\
>>  = 0, queue_desc_len = 0, rss_conf = 0x0, rss_confPA = 0, mf_table = 0x0}
>> 
>> Thanks in advance. 
>> 
>> Dan
>> 
>> On Mar 10, 2014, at 1:00 AM, Prashant Upadhyaya > aricent.com> wrote:
>> 
>>> Hi Srini,
>>> 
>>> Thanks, I could also make it work, thanks to your cue !
>>> 
>>> Now then, this multi-segment not being supported in vmxnet3 driver is a big 
>>> party-pooper for me. Unfortunately in my usecase, I do indeed make heavy 
>>> use of multisegment buffers for sending out the data, so my usecase has 
>>> failed and I will have to fix that.
>>> 
>>> Also, can you please adivse how much is the max data rates you have been 
>>> able to achieve with one vmxnet3 10G port.
>>> 
>>> Thanks a lot for the advice once again.
>>> 
>>> Regards
>>> -Prashant
>>> 
>>> 
>>> -Original Message-
>>> From: Srinivasan J [mailto:srinidpdk at 

[dpdk-dev] On vmxnet-pmd crash in DPDK 1.6.0r1

[Daniel Kan]

Upon further trace, I know what caused it. The uio hardware resources were 
never memory mapped when RTE_EAL_UNBIND_PORTS is not enabled. Specifically, 
pci_dev->mem_resource[] is not mapped.  This explains why setting 
CONFIG_RTE_EAL_UNBIND_PORTS=y fixes the problem. 

lib/librte_pmd_vmxnet3/vmxnet3_ethdev.c
266 static struct eth_driver rte_vmxnet3_pmd = {
267 {
268 .name = "rte_vmxnet3_pmd",
269 .id_table = pci_id_vmxnet3_map,
270 #ifdef RTE_EAL_UNBIND_PORTS
271 .drv_flags = RTE_PCI_DRV_NEED_IGB_UIO,
272 #endif
273 },
274 .eth_dev_init = eth_vmxnet3_dev_init,
275 .dev_private_size = sizeof(struct vmxnet3_adapter),
276 };

Note drv_flags will be 0. 

lib/librte_eal/linuxapp/eal/eal_pci.c
1039 #ifdef RTE_EAL_UNBIND_PORTS
1040 if (dr->drv_flags & RTE_PCI_DRV_NEED_IGB_UIO) {
1041 /* unbind driver and load uio resources for Intel NICs */
1042 if (pci_switch_module(dr, dev, 1, IGB_UIO_NAME) < 0)
...
1050 #else
1051 if (dr->drv_flags & RTE_PCI_DRV_NEED_IGB_UIO)
1052 /* just map resources for Intel NICs */
1053 if (pci_uio_map_resource(dev) < 0)
1054 return -1;
1055 #endif

If RTE_EAL_UNBIND_PORTS  is defined, pci_switch_module will call 
pci_uio_map_resource.

I then looked at the bsdapp?s version, it simply has:
lib/librte_eal/bsdapp/eal/eal_pci.c
479 /* just map the NIC resources */
480 if (pci_uio_map_resource(dev) < 0)
481 return -1;

I don?t know the history behind why .drv_flags = RTE_PCI_DRV_NEED_IGB_UIO only 
for RTE_EAL_UNBIND_PORTS. Can we just ensure pci_uio_map_resource is called in 
linux just like bsd version? One way is to ensure drv_flags is always set to 
RTE_PCI_DRV_NEED_IGB_UIO, but I don?t know if this fix will break other parts. 

+++ vmxnet3_ethdev.c2014-03-10 23:18:02.087742434 -0700
@@ -267,9 +267,7 @@
{
.name = "rte_vmxnet3_pmd",
.id_table = pci_id_vmxnet3_map,
-#ifdef RTE_EAL_UNBIND_PORTS
.drv_flags = RTE_PCI_DRV_NEED_IGB_UIO,
-#endif
},
.eth_dev_init = eth_vmxnet3_dev_init,
.dev_private_size = sizeof(struct vmxnet3_adapter),

Thanks. 

Dan

On Mar 10, 2014, at 10:33 PM, Daniel Kan  wrote:

> I also got the segfault. Setting CONFIG_RTE_EAL_UNBIND_PORTS=y is not ideal 
> because it would also unbind my VMNET3 management interface.
> 
> Does anyone know why the crash is happening? According to the stacktrace, 
> hw_addrX is 0x0 during eth_vmxnet3_dev_init. 
> 
> I?m running on ubuntu 12.04LTS on ESXi 5.5 with Intel I350 NIC as the 
> physical adapter for VMNET3. 
> 
> Program received signal SIGSEGV, Segmentation fault.
> 0x004fe0eb in vmxnet3_read_addr (addr=0x0) at 
> /home/dkan/nyansa/3rd-party/dpdk-1.6.0r1/lib/librte_pmd_vmxnet3/vmxnet3_ethdev.h:139
> (gdb) up
> #1  0x004fe331 in eth_vmxnet3_dev_init (eth_drv=0x799440 
> , eth_dev=0x7d5280 ) at 
> /home/dkan/nyansa/3rd-party/dpdk-1.6.0r1/lib/librte_pmd_v\
> mxnet3/vmxnet3_ethdev.c:218
> (gdb) print *hw
> $12 = {hw_addr0 = 0x0, hw_addr1 = 0x0, back = 0x0, device_id = 1968, 
> vendor_id = 5549, subsystem_device_id = 0, subsystem_vendor_id = 0, 
> adapter_stopped = 0, perm_addr = "\000\000\\
> 000\000\000", num_tx_queues = 1 '\001', num_rx_queues = 1 '\001', 
> bufs_per_pkt = 1 '\001', cur_mtu = 1500, tqd_start = 0x0, rqd_start = 0x0, 
> shared = 0x0, sharedPA = 0, queueDescPA\
>  = 0, queue_desc_len = 0, rss_conf = 0x0, rss_confPA = 0, mf_table = 0x0}
> 
> Thanks in advance. 
> 
> Dan
> 
> On Mar 10, 2014, at 1:00 AM, Prashant Upadhyaya  aricent.com> wrote:
> 
>> Hi Srini,
>> 
>> Thanks, I could also make it work, thanks to your cue !
>> 
>> Now then, this multi-segment not being supported in vmxnet3 driver is a big 
>> party-pooper for me. Unfortunately in my usecase, I do indeed make heavy use 
>> of multisegment buffers for sending out the data, so my usecase has failed 
>> and I will have to fix that.
>> 
>> Also, can you please adivse how much is the max data rates you have been 
>> able to achieve with one vmxnet3 10G port.
>> 
>> Thanks a lot for the advice once again.
>> 
>> Regards
>> -Prashant
>> 
>> 
>> -Original Message-
>> From: Srinivasan J [mailto:srinidpdk at gmail.com]
>> Sent: Sunday, March 09, 2014 12:38 AM
>> To: Prashant Upadhyaya
>> Cc: David Marchand; dev at dpdk.org
>> Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x?
>> 
>> Prashant,
>> I was also able to hit the issue your hitting using Esxi
>> 5.1.0 evaluation and Fedora 20 X86_64 guest. I was able to fix the issue by 
>> setting CONFIG_RTE_EAL_UNBIND_PORTS=y option in 
>> defconfig_x86_64-default-linuxapp-gcc configuration file.
>> 
>> Issue seen
>> 
>> 
>> EAL: PCI device :03:00.0 on NUMA socket -1
>> EAL:   probe driver: 15ad:7b0 rte_vmxnet3_pmd
>> EAL:   Device is blacklisted, not initializing
>> EAL: PCI device :0b:00.0 on NUMA socket -1
>> EAL:   probe