[dpdk-dev] On vmxnet-pmd crash in DPDK 1.6.0r1
10/03/2014 23:29, Daniel Kan :
> virtio-pmd has the same pattern. I wonder if vmxnet3-pmd just blindly copied
> the same paradigm.
>
> 473 static struct eth_driver rte_virtio_pmd = {
> 474 {
> 475 .name = "rte_virtio_pmd",
> 476 .id_table = pci_id_virtio_map,
> 477 #ifdef RTE_EAL_UNBIND_PORTS
> 478 .drv_flags = RTE_PCI_DRV_NEED_IGB_UIO,
> 479 #endif
[...]
> > 266 static struct eth_driver rte_vmxnet3_pmd = {
> > 267 {
> > 268 .name = "rte_vmxnet3_pmd",
> > 269 .id_table = pci_id_vmxnet3_map,
> > 270 #ifdef RTE_EAL_UNBIND_PORTS
> > 271 .drv_flags = RTE_PCI_DRV_NEED_IGB_UIO,
> > 272 #endif
It was the old way of configuring UIO PMD before this patch:
pci: use igb_uio mapping only when needed
http://dpdk.org/browse/dpdk/commit/?id=10ed99419b12b8
The goal of this patch was to fix usage of the extensions virtio-net-pmd and
vmxnet3-usermap.
You can see in this patch that em/igb/ixgbe drivers are updated accordingly.
Then virtio_uio and vmxnet3_uio were released without this patch in mind. So
they were broken.
> > lib/librte_eal/linuxapp/eal/eal_pci.c
> > 1039 #ifdef RTE_EAL_UNBIND_PORTS
> > 1040 if (dr->drv_flags & RTE_PCI_DRV_NEED_IGB_UIO) {
> > 1041 /* unbind driver and load uio resources for Intel NICs */
> > 1042 if (pci_switch_module(dr, dev, 1, IGB_UIO_NAME) < 0)
> > ...
> > 1050 #else
> > 1051 if (dr->drv_flags & RTE_PCI_DRV_NEED_IGB_UIO)
> > 1052 /* just map resources for Intel NICs */
> > 1053 if (pci_uio_map_resource(dev) < 0)
> > 1054 return -1;
> > 1055 #endif
> >
> > If RTE_EAL_UNBIND_PORTS is defined, pci_switch_module will call
> > pci_uio_map_resource.
> >
> > I then looked at the bsdapp?s version, it simply has:
> > lib/librte_eal/bsdapp/eal/eal_pci.c
> > 479 /* just map the NIC resources */
> > 480 if (pci_uio_map_resource(dev) < 0)
> > 481 return -1;
> >
> > I don?t know the history behind why .drv_flags = RTE_PCI_DRV_NEED_IGB_UIO
> > only for RTE_EAL_UNBIND_PORTS. Can we just ensure pci_uio_map_resource is
> > called in linux just like bsd version? One way is to ensure drv_flags is
> > always set to RTE_PCI_DRV_NEED_IGB_UIO, but I don?t know if this fix will
> > break other parts.
I've just sent another patch to update and fix virtio_uio and vmxnet3_uio:
http://dpdk.org/ml/archives/dev/2014-March/001647.html
Feel free to ack it.
Thanks for reporting
--
Thomas
[dpdk-dev] On vmxnet-pmd crash in DPDK 1.6.0r1
virtio-pmd has the same pattern. I wonder if vmxnet3-pmd just blindly copied
the same paradigm.
lib/librte_pmd_virtio/virtio_ethdev.c
473 static struct eth_driver rte_virtio_pmd = {
474 {
475 .name = "rte_virtio_pmd",
476 .id_table = pci_id_virtio_map,
477 #ifdef RTE_EAL_UNBIND_PORTS
478 .drv_flags = RTE_PCI_DRV_NEED_IGB_UIO,
479 #endif
480 },
481 .eth_dev_init = eth_virtio_dev_init,
482 .dev_private_size = sizeof(struct virtio_adapter),
483 };
On Mar 10, 2014, at 11:20 PM, Daniel Kan wrote:
> Upon further trace, I know what caused it. The uio hardware resources were
> never memory mapped when RTE_EAL_UNBIND_PORTS is not enabled. Specifically,
> pci_dev->mem_resource[] is not mapped. This explains why setting
> CONFIG_RTE_EAL_UNBIND_PORTS=y fixes the problem.
>
> lib/librte_pmd_vmxnet3/vmxnet3_ethdev.c
> 266 static struct eth_driver rte_vmxnet3_pmd = {
> 267 {
> 268 .name = "rte_vmxnet3_pmd",
> 269 .id_table = pci_id_vmxnet3_map,
> 270 #ifdef RTE_EAL_UNBIND_PORTS
> 271 .drv_flags = RTE_PCI_DRV_NEED_IGB_UIO,
> 272 #endif
> 273 },
> 274 .eth_dev_init = eth_vmxnet3_dev_init,
> 275 .dev_private_size = sizeof(struct vmxnet3_adapter),
> 276 };
>
> Note drv_flags will be 0.
>
> lib/librte_eal/linuxapp/eal/eal_pci.c
> 1039 #ifdef RTE_EAL_UNBIND_PORTS
> 1040 if (dr->drv_flags & RTE_PCI_DRV_NEED_IGB_UIO) {
> 1041 /* unbind driver and load uio resources for Intel NICs */
> 1042 if (pci_switch_module(dr, dev, 1, IGB_UIO_NAME) < 0)
> ...
> 1050 #else
> 1051 if (dr->drv_flags & RTE_PCI_DRV_NEED_IGB_UIO)
> 1052 /* just map resources for Intel NICs */
> 1053 if (pci_uio_map_resource(dev) < 0)
> 1054 return -1;
> 1055 #endif
>
> If RTE_EAL_UNBIND_PORTS is defined, pci_switch_module will call
> pci_uio_map_resource.
>
> I then looked at the bsdapp?s version, it simply has:
> lib/librte_eal/bsdapp/eal/eal_pci.c
> 479 /* just map the NIC resources */
> 480 if (pci_uio_map_resource(dev) < 0)
> 481 return -1;
>
> I don?t know the history behind why .drv_flags = RTE_PCI_DRV_NEED_IGB_UIO
> only for RTE_EAL_UNBIND_PORTS. Can we just ensure pci_uio_map_resource is
> called in linux just like bsd version? One way is to ensure drv_flags is
> always set to RTE_PCI_DRV_NEED_IGB_UIO, but I don?t know if this fix will
> break other parts.
>
> +++ vmxnet3_ethdev.c 2014-03-10 23:18:02.087742434 -0700
> @@ -267,9 +267,7 @@
> {
> .name = "rte_vmxnet3_pmd",
> .id_table = pci_id_vmxnet3_map,
> -#ifdef RTE_EAL_UNBIND_PORTS
> .drv_flags = RTE_PCI_DRV_NEED_IGB_UIO,
> -#endif
> },
> .eth_dev_init = eth_vmxnet3_dev_init,
> .dev_private_size = sizeof(struct vmxnet3_adapter),
>
> Thanks.
>
> Dan
>
> On Mar 10, 2014, at 10:33 PM, Daniel Kan wrote:
>
>> I also got the segfault. Setting CONFIG_RTE_EAL_UNBIND_PORTS=y is not ideal
>> because it would also unbind my VMNET3 management interface.
>>
>> Does anyone know why the crash is happening? According to the stacktrace,
>> hw_addrX is 0x0 during eth_vmxnet3_dev_init.
>>
>> I?m running on ubuntu 12.04LTS on ESXi 5.5 with Intel I350 NIC as the
>> physical adapter for VMNET3.
>>
>> Program received signal SIGSEGV, Segmentation fault.
>> 0x004fe0eb in vmxnet3_read_addr (addr=0x0) at
>> /home/dkan/nyansa/3rd-party/dpdk-1.6.0r1/lib/librte_pmd_vmxnet3/vmxnet3_ethdev.h:139
>> (gdb) up
>> #1 0x004fe331 in eth_vmxnet3_dev_init (eth_drv=0x799440
>> , eth_dev=0x7d5280 ) at
>> /home/dkan/nyansa/3rd-party/dpdk-1.6.0r1/lib/librte_pmd_v\
>> mxnet3/vmxnet3_ethdev.c:218
>> (gdb) print *hw
>> $12 = {hw_addr0 = 0x0, hw_addr1 = 0x0, back = 0x0, device_id = 1968,
>> vendor_id = 5549, subsystem_device_id = 0, subsystem_vendor_id = 0,
>> adapter_stopped = 0, perm_addr = "\000\000\\
>> 000\000\000", num_tx_queues = 1 '\001', num_rx_queues = 1 '\001',
>> bufs_per_pkt = 1 '\001', cur_mtu = 1500, tqd_start = 0x0, rqd_start = 0x0,
>> shared = 0x0, sharedPA = 0, queueDescPA\
>> = 0, queue_desc_len = 0, rss_conf = 0x0, rss_confPA = 0, mf_table = 0x0}
>>
>> Thanks in advance.
>>
>> Dan
>>
>> On Mar 10, 2014, at 1:00 AM, Prashant Upadhyaya > aricent.com> wrote:
>>
>>> Hi Srini,
>>>
>>> Thanks, I could also make it work, thanks to your cue !
>>>
>>> Now then, this multi-segment not being supported in vmxnet3 driver is a big
>>> party-pooper for me. Unfortunately in my usecase, I do indeed make heavy
>>> use of multisegment buffers for sending out the data, so my usecase has
>>> failed and I will have to fix that.
>>>
>>> Also, can you please adivse how much is the max data rates you have been
>>> able to achieve with one vmxnet3 10G port.
>>>
>>> Thanks a lot for the advice once again.
>>>
>>> Regards
>>> -Prashant
>>>
>>>
>>> -Original Message-
>>> From: Srinivasan J [mailto:srinidpdk at
[dpdk-dev] On vmxnet-pmd crash in DPDK 1.6.0r1
Upon further trace, I know what caused it. The uio hardware resources were
never memory mapped when RTE_EAL_UNBIND_PORTS is not enabled. Specifically,
pci_dev->mem_resource[] is not mapped. This explains why setting
CONFIG_RTE_EAL_UNBIND_PORTS=y fixes the problem.
lib/librte_pmd_vmxnet3/vmxnet3_ethdev.c
266 static struct eth_driver rte_vmxnet3_pmd = {
267 {
268 .name = "rte_vmxnet3_pmd",
269 .id_table = pci_id_vmxnet3_map,
270 #ifdef RTE_EAL_UNBIND_PORTS
271 .drv_flags = RTE_PCI_DRV_NEED_IGB_UIO,
272 #endif
273 },
274 .eth_dev_init = eth_vmxnet3_dev_init,
275 .dev_private_size = sizeof(struct vmxnet3_adapter),
276 };
Note drv_flags will be 0.
lib/librte_eal/linuxapp/eal/eal_pci.c
1039 #ifdef RTE_EAL_UNBIND_PORTS
1040 if (dr->drv_flags & RTE_PCI_DRV_NEED_IGB_UIO) {
1041 /* unbind driver and load uio resources for Intel NICs */
1042 if (pci_switch_module(dr, dev, 1, IGB_UIO_NAME) < 0)
...
1050 #else
1051 if (dr->drv_flags & RTE_PCI_DRV_NEED_IGB_UIO)
1052 /* just map resources for Intel NICs */
1053 if (pci_uio_map_resource(dev) < 0)
1054 return -1;
1055 #endif
If RTE_EAL_UNBIND_PORTS is defined, pci_switch_module will call
pci_uio_map_resource.
I then looked at the bsdapp?s version, it simply has:
lib/librte_eal/bsdapp/eal/eal_pci.c
479 /* just map the NIC resources */
480 if (pci_uio_map_resource(dev) < 0)
481 return -1;
I don?t know the history behind why .drv_flags = RTE_PCI_DRV_NEED_IGB_UIO only
for RTE_EAL_UNBIND_PORTS. Can we just ensure pci_uio_map_resource is called in
linux just like bsd version? One way is to ensure drv_flags is always set to
RTE_PCI_DRV_NEED_IGB_UIO, but I don?t know if this fix will break other parts.
+++ vmxnet3_ethdev.c2014-03-10 23:18:02.087742434 -0700
@@ -267,9 +267,7 @@
{
.name = "rte_vmxnet3_pmd",
.id_table = pci_id_vmxnet3_map,
-#ifdef RTE_EAL_UNBIND_PORTS
.drv_flags = RTE_PCI_DRV_NEED_IGB_UIO,
-#endif
},
.eth_dev_init = eth_vmxnet3_dev_init,
.dev_private_size = sizeof(struct vmxnet3_adapter),
Thanks.
Dan
On Mar 10, 2014, at 10:33 PM, Daniel Kan wrote:
> I also got the segfault. Setting CONFIG_RTE_EAL_UNBIND_PORTS=y is not ideal
> because it would also unbind my VMNET3 management interface.
>
> Does anyone know why the crash is happening? According to the stacktrace,
> hw_addrX is 0x0 during eth_vmxnet3_dev_init.
>
> I?m running on ubuntu 12.04LTS on ESXi 5.5 with Intel I350 NIC as the
> physical adapter for VMNET3.
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x004fe0eb in vmxnet3_read_addr (addr=0x0) at
> /home/dkan/nyansa/3rd-party/dpdk-1.6.0r1/lib/librte_pmd_vmxnet3/vmxnet3_ethdev.h:139
> (gdb) up
> #1 0x004fe331 in eth_vmxnet3_dev_init (eth_drv=0x799440
> , eth_dev=0x7d5280 ) at
> /home/dkan/nyansa/3rd-party/dpdk-1.6.0r1/lib/librte_pmd_v\
> mxnet3/vmxnet3_ethdev.c:218
> (gdb) print *hw
> $12 = {hw_addr0 = 0x0, hw_addr1 = 0x0, back = 0x0, device_id = 1968,
> vendor_id = 5549, subsystem_device_id = 0, subsystem_vendor_id = 0,
> adapter_stopped = 0, perm_addr = "\000\000\\
> 000\000\000", num_tx_queues = 1 '\001', num_rx_queues = 1 '\001',
> bufs_per_pkt = 1 '\001', cur_mtu = 1500, tqd_start = 0x0, rqd_start = 0x0,
> shared = 0x0, sharedPA = 0, queueDescPA\
> = 0, queue_desc_len = 0, rss_conf = 0x0, rss_confPA = 0, mf_table = 0x0}
>
> Thanks in advance.
>
> Dan
>
> On Mar 10, 2014, at 1:00 AM, Prashant Upadhyaya aricent.com> wrote:
>
>> Hi Srini,
>>
>> Thanks, I could also make it work, thanks to your cue !
>>
>> Now then, this multi-segment not being supported in vmxnet3 driver is a big
>> party-pooper for me. Unfortunately in my usecase, I do indeed make heavy use
>> of multisegment buffers for sending out the data, so my usecase has failed
>> and I will have to fix that.
>>
>> Also, can you please adivse how much is the max data rates you have been
>> able to achieve with one vmxnet3 10G port.
>>
>> Thanks a lot for the advice once again.
>>
>> Regards
>> -Prashant
>>
>>
>> -Original Message-
>> From: Srinivasan J [mailto:srinidpdk at gmail.com]
>> Sent: Sunday, March 09, 2014 12:38 AM
>> To: Prashant Upadhyaya
>> Cc: David Marchand; dev at dpdk.org
>> Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x?
>>
>> Prashant,
>> I was also able to hit the issue your hitting using Esxi
>> 5.1.0 evaluation and Fedora 20 X86_64 guest. I was able to fix the issue by
>> setting CONFIG_RTE_EAL_UNBIND_PORTS=y option in
>> defconfig_x86_64-default-linuxapp-gcc configuration file.
>>
>> Issue seen
>>
>>
>> EAL: PCI device :03:00.0 on NUMA socket -1
>> EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd
>> EAL: Device is blacklisted, not initializing
>> EAL: PCI device :0b:00.0 on NUMA socket -1
>> EAL: probe
[dpdk-dev] On vmxnet-pmd crash in DPDK 1.6.0r1
I also got the segfault. Setting CONFIG_RTE_EAL_UNBIND_PORTS=y is not ideal
because it would also unbind my VMNET3 management interface.
Does anyone know why the crash is happening? According to the stacktrace,
hw_addrX is 0x0 during eth_vmxnet3_dev_init.
I?m running on ubuntu 12.04LTS on ESXi 5.5 with Intel I350 NIC as the physical
adapter for VMNET3.
Program received signal SIGSEGV, Segmentation fault.
0x004fe0eb in vmxnet3_read_addr (addr=0x0) at
/home/dkan/nyansa/3rd-party/dpdk-1.6.0r1/lib/librte_pmd_vmxnet3/vmxnet3_ethdev.h:139
(gdb) up
#1 0x004fe331 in eth_vmxnet3_dev_init (eth_drv=0x799440
, eth_dev=0x7d5280 ) at
/home/dkan/nyansa/3rd-party/dpdk-1.6.0r1/lib/librte_pmd_v\
mxnet3/vmxnet3_ethdev.c:218
(gdb) print *hw
$12 = {hw_addr0 = 0x0, hw_addr1 = 0x0, back = 0x0, device_id = 1968, vendor_id
= 5549, subsystem_device_id = 0, subsystem_vendor_id = 0, adapter_stopped = 0,
perm_addr = "\000\000\\
000\000\000", num_tx_queues = 1 '\001', num_rx_queues = 1 '\001', bufs_per_pkt
= 1 '\001', cur_mtu = 1500, tqd_start = 0x0, rqd_start = 0x0, shared = 0x0,
sharedPA = 0, queueDescPA\
= 0, queue_desc_len = 0, rss_conf = 0x0, rss_confPA = 0, mf_table = 0x0}
Thanks in advance.
Dan
On Mar 10, 2014, at 1:00 AM, Prashant Upadhyaya wrote:
> Hi Srini,
>
> Thanks, I could also make it work, thanks to your cue !
>
> Now then, this multi-segment not being supported in vmxnet3 driver is a big
> party-pooper for me. Unfortunately in my usecase, I do indeed make heavy use
> of multisegment buffers for sending out the data, so my usecase has failed
> and I will have to fix that.
>
> Also, can you please adivse how much is the max data rates you have been able
> to achieve with one vmxnet3 10G port.
>
> Thanks a lot for the advice once again.
>
> Regards
> -Prashant
>
>
> -Original Message-
> From: Srinivasan J [mailto:srinidpdk at gmail.com]
> Sent: Sunday, March 09, 2014 12:38 AM
> To: Prashant Upadhyaya
> Cc: David Marchand; dev at dpdk.org
> Subject: Re: [dpdk-dev] Which vmxnet3 pmd is to be used in dpdk 1.6.x?
>
> Prashant,
> I was also able to hit the issue your hitting using Esxi
> 5.1.0 evaluation and Fedora 20 X86_64 guest. I was able to fix the issue by
> setting CONFIG_RTE_EAL_UNBIND_PORTS=y option in
> defconfig_x86_64-default-linuxapp-gcc configuration file.
>
> Issue seen
>
>
> EAL: PCI device :03:00.0 on NUMA socket -1
> EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd
> EAL: Device is blacklisted, not initializing
> EAL: PCI device :0b:00.0 on NUMA socket -1
> EAL: probe driver: 15ad:7b0 rte_vmxnet3_pmd
>
> Program received signal SIGSEGV, Segmentation fault.
> eth_vmxnet3_dev_init (eth_drv=, eth_dev=0x754480
> ) at
> /root/source/dpdk-1.6.0r1/lib/librte_pmd_vmxnet3/vmxnet3_ethdev.c:218
> 218 ver = VMXNET3_READ_BAR1_REG(hw, VMXNET3_REG_VRRS);
> Missing separate debuginfos, use: debuginfo-install glibc-2.18-11.fc20.x86_64
> (gdb) p hw
> $1 = (struct vmxnet3_hw *) 0x7fffd8fc1040
> (gdb) p *hw
> $2 = {hw_addr0 = 0x0, hw_addr1 = 0x0, back = 0x0, device_id = 1968, vendor_id
> = 5549, subsystem_device_id = 0, subsystem_vendor_id = 0,
> adapter_stopped = 0, perm_addr = "\000\000\000\000\000", num_tx_queues = 1
> '\001', num_rx_queues = 1 '\001', bufs_per_pkt = 1 '\001', cur_mtu = 0,
> tqd_start = 0x0, rqd_start = 0x0, shared = 0x0, sharedPA = 0, queueDescPA =
> 0, queue_desc_len = 0, rss_conf = 0x0, rss_confPA = 0, mf_table = 0x0}
> (gdb)
>
> #define VMXNET3_PCI_BAR1_REG_ADDR(hw, reg) \
>((volatile uint32_t *)((char *)(hw)->hw_addr1 + (reg))) #define
> VMXNET3_READ_BAR1_REG(hw, reg) \
>vmxnet3_read_addr(VMXNET3_PCI_BAR1_REG_ADDR((hw), (reg)))
> lib/librte_pmd_vmxnet3/vmxnet3_ethdev.h
>
> Issue not seen after enabling CONFIG_RTE_EAL_UNBIND_PORTS=y
> ==
>
> [root at localhost build]# ./l2fwd -c 0xf -b :03:00.0 -n 1 -- -p 0x6
> EAL: Detected lcore 0 as core 0 on socket 0
> EAL: Detected lcore 1 as core 1 on socket 0
> EAL: Detected lcore 2 as core 2 on socket 0
> EAL: Detected lcore 3 as core 3 on socket 0
> EAL: Skip lcore 4 (not detected)
> EAL: Skip lcore 5 (not detected)
> EAL: Skip lcore 6 (not detected)
> EAL: Skip lcore 7 (not detected)
> EAL: Skip lcore 8 (not detected)
> EAL: Skip lcore 9 (not detected)
> EAL: Skip lcore 10 (not detected)
> EAL: Skip lcore 11 (not detected)
> EAL: Skip lcore 12 (not detected)
> EAL: Skip lcore 13 (not detected)
> EAL: Skip lcore 14 (not detected)
> EAL: Skip lcore 15 (not detected)
> EAL: Skip lcore 16 (not detected)
> EAL: Skip lcore 17 (not detected)
> EAL: Skip lcore 18 (not detected)
> EAL: Skip lcore 19 (not detected)
> EAL: Skip lcore 20 (not detected)
> EAL: Skip lcore 21 (not detected)
> EAL: Skip lcore 22 (not detected)
> EAL: Skip lcore 23 (not detected)
> EAL: Skip lcore 24 (not detected)
> EAL: Skip lcore 25 (not detected)
> EAL: Skip lcore 26 (not detected)
> EAL: Skip lcore 27 (not
