[dpdk-dev] dpdk 16.07, issues with rte_mempool_create and rte_kni_alloc()
Thank you Ferruh, I will give this a spin over the weekend and let you know. Rgds, Gopa. On Thu, Aug 25, 2016 at 6:51 AM, Ferruh Yigit wrote: > On 8/10/2016 11:51 AM, Ferruh Yigit wrote: > > Hi Gopakumar, > > > > On 8/4/2016 5:14 PM, Ferruh Yigit wrote: > >> On 8/1/2016 10:19 PM, Gopakumar Choorakkot Edakkunni wrote: > >>> Well, for my purpose I just ended up creating a seperate/smaller pool > >>> earlier during bootup to try to guarantee its from one memseg. > >>> > >>> But I am assuming that this KNI restriction is something thats > "currently" > >>> not fixed and is "fixable" ? > >> > >> > >>> Any ideas on what the summary of the reason > >>> for this restriction is - I was gonna check if I can fix that > >> > >> KNI expects all mbufs are from a physically continuous memory. This is > >> because of current address translation implementation. > >> > >> mbufs allocated in userspace and accessed from both user and kernel > >> space, so mbuf userspace virtual address needs to be converted into > >> kernelspace virtual address. > >> > >> Currently this address translation done by first calculating an offset > >> between virtual addresses using first filed of mempool, later applying > >> same offset to all mbufs. This is why all mbufs should be in physically > >> continuous memory. > >> > >> I think this address translation can be done in different way which can > >> remove the restriction, but not sure about the effect of the > >> performance. I will send a patch for this. > > > > I have sent a patch to remove KNI restriction: > > http://dpdk.org/dev/patchwork/patch/15171/ > > > > Can you please test this patch with a mempool with multiple memzone? > > You need to remove following check in KNI manually: > > if (mp->nb_mem_chunks != 1) > > goto kni_fail; > > Hi Gopakumar, > > Off the list. > > Any chance to test this? > > Thanks, > ferruh > >
[dpdk-dev] SR-IOV: dpdk16.07 on guest and ixgbe 3.19.1 on host - will that be a problem ? (requested invalid api version)
I see the below message "requested invalid api version" on host ? Will that be an issue or is it benign ? Should I upgrade the host ixgbe to a later version ? On the host: *[69576.199458] ixgbe :88:00.1 p4p2: VF 2 requested invalid api version 3* root# lsb_release -a No LSB modules are available. Distributor ID:Ubuntu Description:Ubuntu 14.04.4 LTS Release:14.04 Codename:trusty root at mtpnjrsv113:~# root# ethtool -i p4p2 driver: ixgbe version: 3.19.1 firmware-version: 0x8811, 1.1067.0 bus-info: :88:00.1 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: no On the guest: root# lsb_release -a No LSB modules are available. Distributor ID:Ubuntu Description:Ubuntu 14.04.2 LTS Release:14.04 Codename:trusty root# find / -name rte_kni.ko /var/lib/dkms/dpdk/16.07/3.16.0-30-generic/x86_64/module/rte_kni.ko /lib/modules/3.16.0-30-generic/updates/dkms/rte_kni.ko
[dpdk-dev] dpdk 16.07, issues with rte_mempool_create and rte_kni_alloc()
Well, for my purpose I just ended up creating a seperate/smaller pool earlier during bootup to try to guarantee its from one memseg. But I am assuming that this KNI restriction is something thats "currently" not fixed and is "fixable" ? Any ideas on what the summary of the reason for this restriction is - I was gonna check if I can fix that Rgds, Gopa. On Sat, Jul 30, 2016 at 10:39 PM, Gopakumar Choorakkot Edakkunni < gopakumar.c.e at gmail.com> wrote: > Hi all, > > So 16.07 seems to have an rte_mempool_create() which can allocate items > from multiple memzones (commit d1d914eb), which is awesome and is the > reason I upgraed to 16.07 !! > > But .. when I call rte_kni_alloc() with the mempool as parameter, theres a > check in there which says below > > /* KNI currently requires to have only one memory chunk */ > if (mp->nb_mem_chunks != 1) > goto kni_fail; > > And I hit that check because I am allocating a large mempool that happens > to span multiple memzones. I am perfectly fine with allocating a seperate > mempool just for KNI, except that I am unable to find any API or a flag or > some setting which says "create me a mempool that doesnt span memzones", > just so that I can use it with rte_kni_alloc() > > Any suggestions will be helpful ! > > Rgds, > Gopa. > >
[dpdk-dev] dpdk 16.07, issues with rte_mempool_create and rte_kni_alloc()
Hi all, So 16.07 seems to have an rte_mempool_create() which can allocate items from multiple memzones (commit d1d914eb), which is awesome and is the reason I upgraed to 16.07 !! But .. when I call rte_kni_alloc() with the mempool as parameter, theres a check in there which says below /* KNI currently requires to have only one memory chunk */ if (mp->nb_mem_chunks != 1) goto kni_fail; And I hit that check because I am allocating a large mempool that happens to span multiple memzones. I am perfectly fine with allocating a seperate mempool just for KNI, except that I am unable to find any API or a flag or some setting which says "create me a mempool that doesnt span memzones", just so that I can use it with rte_kni_alloc() Any suggestions will be helpful ! Rgds, Gopa.
[dpdk-dev] ethtool doesnt work on some interface after unbinding dpdk
Just to update this thread. With power management for IGB disabled, this problem is not seen .. So for now thats the "workaround" Rgds, Gopa. On Mon, Apr 18, 2016 at 3:08 AM, Remy Horton wrote: > > On 15/04/2016 23:56, Gopakumar Choorakkot Edakkunni wrote: > >> This time the problem statement is more narrowed down. >> >> 1. dpdk is enabled on the interface, interfaces bound to igb_uio >> 3. kill the process using dpdk >> 3. rmmod rte_kni >> 4. rmmod igb_uio >> 5. bind interface to igb >> 6. ethtool, ifconfig up/down etc.. works for approximately 30 seconds, >> and then stops working >> > > Hmm.. can you try that but with rte_kni left out completely? KNI hooks > into the Linux network stack and think it at the least needs eliminating as > a casual factor. Can you also try using uio_pci_generic rather than igb_uio? > > Those aside, I'm suspecting driver issues, so seeing if I can get one of > the driver test guys to have a look at this.. > > > Regards, > > ..Remy >
[dpdk-dev] ethtool doesnt work on some interface after unbinding dpdk
Well, I jumped to a conclusion too soon on the ACPI, that was a wrong statement (wishful thinking), I recreated the issue even with ACPI off This time the problem statement is more narrowed down. 1. dpdk is enabled on the interface, interfaces bound to igb_uio 3. kill the process using dpdk 3. rmmod rte_kni 4. rmmod igb_uio 5. bind interface to igb 6. ethtool, ifconfig up/down etc.. works for approximately 30 seconds, and then stops working At step #6, if I do an lspci of the device, the device is completely shut down (attached a sample lspci output - memory not initialized, irqs not present in /proc/interrupts etc..). But theres nothing in the dmesg that shows any kind of errors after the messages about the interface being bound in step #5 And the wierd part is that the device is up at step #6 for like 30 seconds before it appears shut down in the pci output. Another observation is that in step #4 when igb_uio relinquishes control of the device it still seems to leave it in an initialized state, so the only theory I can think of is whether the device being left in an initia;ized state when it was handed off to igb in step #5 caused igb to run into some error and shut down the device ? The next step is to enable debugs in igb driver and see if there are any debugs that tells more about what happened Rgds, Gopa. On Fri, Apr 15, 2016 at 12:31 PM, Gopakumar Choorakkot Edakkunni < gopakumar.c.e at gmail.com> wrote: > So looks like I figured it out .. I came across this bug reference > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=728692 and thought of > checking out my problem with ACPI turned off. And with ACPI turned off, the > problem doesnt happen. So theres something that the igb driver is not happy > about when acpi is on .. Any thoughts ? > > Rgds, > Gopa. > > On Fri, Apr 15, 2016 at 10:38 AM, Gopakumar Choorakkot Edakkunni < > gopakumar.c.e at gmail.com> wrote: > >> Nothing in dmesg .. The ethtool was just a side-observation, the biggest >> problem was that after unbinding from igb_uio and rebinding to igb, if I >> follow it up with an /etc/init.d/network restart, that completely hoses the >> linux system - anyone trying to open a socket (ifconfig for example) just >> hangs. Thats how I started troubleshooting this and happened to see this >> ethtool thing along with it, not sure if its related. Also the issue >> doesn't happen with one or two interfaces, there needs to be at least five >> or six interfaces for this to happen. >> >> The other thing I noticed is that if I put some sleep (2 seconds) between >> unbind igb_uio and re-bind igb, the network-restart-hosing-system doesnt >> happen, but the ethtool issue still remains >> >> Rgds, >> Gopa. >> >> On Fri, Apr 15, 2016 at 12:50 AM, Remy Horton >> wrote: >> >>> On 14/04/2016 20:25, Gopakumar Choorakkot Edakkunni wrote: >>> [..] >>> >>>> ge8-> 06:00.0 Ethernet controller: Intel Corporation 82576 Gigabit >>>> Network Connection (rev 01) >>>> >>>> root:~# ls /sys/class/net/ge8/device/driver/module/drivers/ >>>> pci:igb >>>> root:~# >>>> >>>> root:~# ethtool ge8 >>>> Settings for ge8: >>>> Cannot get device settings: No such device >>>> Cannot get wake-on-lan settings: No such device >>>> Cannot get message level: No such device >>>> Cannot get link status: No such device >>>> No data available >>>> >>> >>> Seems a little odd. Does dmesg show anything related to igb/ixgbe when >>> you try this? >>> >>> ..Remy >>> >> >> >
[dpdk-dev] ethtool doesnt work on some interface after unbinding dpdk
So looks like I figured it out .. I came across this bug reference https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=728692 and thought of checking out my problem with ACPI turned off. And with ACPI turned off, the problem doesnt happen. So theres something that the igb driver is not happy about when acpi is on .. Any thoughts ? Rgds, Gopa. On Fri, Apr 15, 2016 at 10:38 AM, Gopakumar Choorakkot Edakkunni < gopakumar.c.e at gmail.com> wrote: > Nothing in dmesg .. The ethtool was just a side-observation, the biggest > problem was that after unbinding from igb_uio and rebinding to igb, if I > follow it up with an /etc/init.d/network restart, that completely hoses the > linux system - anyone trying to open a socket (ifconfig for example) just > hangs. Thats how I started troubleshooting this and happened to see this > ethtool thing along with it, not sure if its related. Also the issue > doesn't happen with one or two interfaces, there needs to be at least five > or six interfaces for this to happen. > > The other thing I noticed is that if I put some sleep (2 seconds) between > unbind igb_uio and re-bind igb, the network-restart-hosing-system doesnt > happen, but the ethtool issue still remains > > Rgds, > Gopa. > > On Fri, Apr 15, 2016 at 12:50 AM, Remy Horton > wrote: > >> On 14/04/2016 20:25, Gopakumar Choorakkot Edakkunni wrote: >> [..] >> >>> ge8-> 06:00.0 Ethernet controller: Intel Corporation 82576 Gigabit >>> Network Connection (rev 01) >>> >>> root:~# ls /sys/class/net/ge8/device/driver/module/drivers/ >>> pci:igb >>> root:~# >>> >>> root:~# ethtool ge8 >>> Settings for ge8: >>> Cannot get device settings: No such device >>> Cannot get wake-on-lan settings: No such device >>> Cannot get message level: No such device >>> Cannot get link status: No such device >>> No data available >>> >> >> Seems a little odd. Does dmesg show anything related to igb/ixgbe when >> you try this? >> >> ..Remy >> > >
[dpdk-dev] ethtool doesnt work on some interface after unbinding dpdk
Nothing in dmesg .. The ethtool was just a side-observation, the biggest problem was that after unbinding from igb_uio and rebinding to igb, if I follow it up with an /etc/init.d/network restart, that completely hoses the linux system - anyone trying to open a socket (ifconfig for example) just hangs. Thats how I started troubleshooting this and happened to see this ethtool thing along with it, not sure if its related. Also the issue doesn't happen with one or two interfaces, there needs to be at least five or six interfaces for this to happen. The other thing I noticed is that if I put some sleep (2 seconds) between unbind igb_uio and re-bind igb, the network-restart-hosing-system doesnt happen, but the ethtool issue still remains Rgds, Gopa. On Fri, Apr 15, 2016 at 12:50 AM, Remy Horton wrote: > On 14/04/2016 20:25, Gopakumar Choorakkot Edakkunni wrote: > [..] > >> ge8-> 06:00.0 Ethernet controller: Intel Corporation 82576 Gigabit >> Network Connection (rev 01) >> >> root:~# ls /sys/class/net/ge8/device/driver/module/drivers/ >> pci:igb >> root:~# >> >> root:~# ethtool ge8 >> Settings for ge8: >> Cannot get device settings: No such device >> Cannot get wake-on-lan settings: No such device >> Cannot get message level: No such device >> Cannot get link status: No such device >> No data available >> > > Seems a little odd. Does dmesg show anything related to igb/ixgbe when you > try this? > > ..Remy >
[dpdk-dev] ethtool doesnt work on some interface after unbinding dpdk
Hi Remy, Thanks for the response. The error is "No such device", some snippets below. And no I was not using the dpdk ethtool ge8-> 06:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) root:~# ls /sys/class/net/ge8/device/driver/module/drivers/ pci:igb root:~# root:~# ethtool ge8 Settings for ge8: Cannot get device settings: No such device Cannot get wake-on-lan settings: No such device Cannot get message level: No such device Cannot get link status: No such device No data available Rgds, Gopa. On Thu, Apr 14, 2016 at 12:15 AM, Remy Horton wrote: > Morning, > > On 13/04/2016 15:48, Gopakumar Choorakkot Edakkunni wrote: > [..] > >> then after a while I >> unbind from igb_uio and bind them back to igb/ixgbe. At this point, one of >> the 4 igb ports (random) stops responding to ethtool, ethtool bails out >> with some error. But otherwise the interface seems to work fine, it has a >> linux interface created and pops up in /sys/class/net etc.. Has anyone >> seen >> this before ? I thought of checking before starting to debug this further >> > > Can you give details of the error? If you were you unbinding from igb_uio > while examples/ethtool was still running it likley caused something to trip > up, as at least DPDK ethtool itself was not made with run-time unbinding in > mind. > > Regards, > > ..Remy >
[dpdk-dev] ethtool doesnt work on some interface after unbinding dpdk
All, I have a Xeon machine with linux kernel 3.14 and dpdk 2.0, I have 6 ports of which 4 are igb and 2 are ixgbe drivers .. I use the python script in the dpdk distribution to bind and unbind the ports. I unbind them from linux and bind to igb_uio, dpdk works just fine, then after a while I unbind from igb_uio and bind them back to igb/ixgbe. At this point, one of the 4 igb ports (random) stops responding to ethtool, ethtool bails out with some error. But otherwise the interface seems to work fine, it has a linux interface created and pops up in /sys/class/net etc.. Has anyone seen this before ? I thought of checking before starting to debug this further Rgds, Gopa.
[dpdk-dev] Performance of rte_ring APIs
Hi All, I have an extremely simple test - I have just one single DPDK EAL thread which pulls packets from one 10G port and just puts the packet exactly as is (no changes) on another 10G port - I get 9.5million pps, so far so good. So its like this dpdk_rx dpdk_tx 9.5 Millionpps Now I do the below and the performance comes down to like 4 Million pps, less than half ! dpdk_rx rte_ring_mc_dequeue_bulk(my_ring1, my_array, nb_rx) dpdk_tx rte_ring_mp_enqueue_bulk(my_ring1, my_array, nb_rx) Note that I do nothing with the things I dequeue from the ring, I just enqueue it back. So is there any gotchas in using these rings ? I am sure I am missing something, it cant drop from 9.5Mpps to 4Mpps just because of a dequeue/enqueue ? Rgds, Gopa.
[dpdk-dev] Diff for persistent KNI interfaces
I guess theres not much interest from others for this support in
general, but putting it out here in case anyone wants to take a look.
If anyone finds any major issues with the diffset, please do let me
know, that will be of great help. I have tested this on my machine.
Rgds,
Gopa.
-- next part --
commit 58968f690056d5afa7517e724bc91f6436639d10
Author: Gopakumar C E
Date: Sun Jul 12 11:18:33 2015 -0700
KNI persistent interfaces
diff --git a/lib/librte_eal/linuxapp/eal/include/exec-env/rte_kni_common.h
b/lib/librte_eal/linuxapp/eal/include/exec-env/rte_kni_common.h
index 1e55c2d..369ab85 100644
--- a/lib/librte_eal/linuxapp/eal/include/exec-env/rte_kni_common.h
+++ b/lib/librte_eal/linuxapp/eal/include/exec-env/rte_kni_common.h
@@ -160,6 +160,7 @@ struct rte_kni_device_info {
uint32_t core_id; /**< core ID to bind for kernel thread */
uint8_t force_bind : 1; /**< Flag for kernel thread binding */
+ uint8_t persist_on_close : 1; /**< Dont delete interface on dev/kni
close */
/* mbuf size */
unsigned mbuf_size;
diff --git a/lib/librte_eal/linuxapp/kni/kni_dev.h
b/lib/librte_eal/linuxapp/kni/kni_dev.h
index e79e472..e50aad7 100644
--- a/lib/librte_eal/linuxapp/kni/kni_dev.h
+++ b/lib/librte_eal/linuxapp/kni/kni_dev.h
@@ -96,6 +96,12 @@ struct kni_dev {
/* synchro for request processing */
unsigned long synchro;
+ /* Persist netdev on /dev/kni close */
+ uint8_t netdev_persist : 1;
+
+ /* /dev/kni is closed or not */
+ uint8_t kni_released : 1;
+
#ifdef RTE_KNI_VHOST
struct kni_vhost_queue* vhost_queue;
volatile enum {
diff --git a/lib/librte_eal/linuxapp/kni/kni_misc.c
b/lib/librte_eal/linuxapp/kni/kni_misc.c
index 1935d32..f386a31 100644
--- a/lib/librte_eal/linuxapp/kni/kni_misc.c
+++ b/lib/librte_eal/linuxapp/kni/kni_misc.c
@@ -58,7 +58,7 @@ static int kni_ioctl(struct inode *inode, unsigned int
ioctl_num,
unsigned long ioctl_param);
static int kni_compat_ioctl(struct inode *inode, unsigned int ioctl_num,
unsigned long ioctl_param);
-static int kni_dev_remove(struct kni_dev *dev);
+static int kni_dev_remove(struct kni_dev *dev, int netdev_remove);
static int __init kni_parse_kthread_mode(void);
@@ -128,6 +128,21 @@ kni_init(void)
static void __exit
kni_exit(void)
{
+struct kni_dev *dev, *n;
+
+ /*
+* Remove all the persistent KNI interfaces.
+*/
+down_write(_list_lock);
+list_for_each_entry_safe(dev, n, _list_head, list) {
+ if (dev->net_dev) {
+ unregister_netdev(dev->net_dev);
+ free_netdev(dev->net_dev);
+ }
+list_del(>list);
+ }
+up_write(_list_lock);
+
misc_deregister(_misc);
KNI_PRINT("### DPDK kni module unloaded ###\n");
}
@@ -196,8 +211,12 @@ kni_release(struct inode *inode, struct file *file)
#ifdef RTE_KNI_VHOST
kni_vhost_backend_release(dev);
#endif
- kni_dev_remove(dev);
- list_del(>list);
+ kni_dev_remove(dev, !dev->netdev_persist);
+ if (!dev->netdev_persist) {
+ list_del(>list);
+ } else {
+ dev->kni_released = 1;
+ }
}
up_write(_list_lock);
@@ -264,7 +283,7 @@ kni_thread_multiple(void *param)
}
static int
-kni_dev_remove(struct kni_dev *dev)
+kni_dev_remove(struct kni_dev *dev, int netdev_remove)
{
if (!dev)
return -ENODEV;
@@ -282,7 +301,7 @@ kni_dev_remove(struct kni_dev *dev)
break;
}
- if (dev->net_dev) {
+ if (dev->net_dev && netdev_remove) {
unregister_netdev(dev->net_dev);
free_netdev(dev->net_dev);
}
@@ -314,8 +333,9 @@ kni_ioctl_create(unsigned int ioctl_num, unsigned long
ioctl_param)
struct pci_dev *found_pci = NULL;
struct net_device *net_dev = NULL;
struct net_device *lad_dev = NULL;
- struct kni_dev *kni, *dev, *n;
+ struct kni_dev *kni = NULL, *dev, *n;
struct net *net;
+ int kni_exists = 0;
printk(KERN_INFO "KNI: Creating kni...\n");
/* Check the buffer size, to avoid warning */
@@ -343,32 +363,52 @@ kni_ioctl_create(unsigned int ioctl_num, unsigned long
ioctl_param)
down_read(_list_lock);
list_for_each_entry_safe(dev, n, _list_head, list) {
if (kni_check_param(dev, _info) < 0) {
- up_read(_list_lock);
- return -EINVAL;
+ kni = dev;
+ kni_exists = 1;
+ break;
}
}
up_read(_list_lock);
- net_dev =
[dpdk-dev] How to prevent KNI interface from getting deleted on application termination?
You are right, I dint think of the rings. I guess if I keep the netdev intact and release the rings and stuff as it does on a /dev/kni close, that should be fine ? And when my app comes back up and opens dev/kni, the rings etc.. can be recreated, and when my app tries to create the netdevs, it reattaches it to the existing ones. The way I look at it, its similar to the tun/tap ioctl option TUNSETPERSIST. Rgds, Gopa. On Thu, Jul 9, 2015 at 9:46 AM, Stephen Hemminger wrote: > On Wed, 8 Jul 2015 23:36:52 -0700 > Gopakumar Choorakkot Edakkunni wrote: > >> Reading through the KNI module source, doesnt look like there is a way >> to do this. For my requirement, I will make some patch tomorrow to >> have a module option to just keep the KNI data structures around even >> if /dev/kni is closed, looks straightforward to do from the code > > > I don't think it is that simple given the shared ring in KNI
[dpdk-dev] How to prevent KNI interface from getting deleted on application termination?
Crash is just one example I called out, the other example is a hotfix update to the app which ends up gracefully restarting the app. Rgds, Gopa. On Thu, Jul 9, 2015 at 12:47 AM, Marc Sune wrote: > > > On 09/07/15 08:36, Gopakumar Choorakkot Edakkunni wrote: >> >> Reading through the KNI module source, doesnt look like there is a way >> to do this. For my requirement, I will make some patch tomorrow to >> have a module option to just keep the KNI data structures around even >> if /dev/kni is closed, looks straightforward to do from the code >> >> Rgds, >> Gopa. >> >> On Wed, Jul 8, 2015 at 1:00 PM, Gopakumar Choorakkot Edakkunni >> wrote: >>> >>> Hi all, >>> >>> My application takes over one/multiple ethernet port(s) in a linux >>> system and creates KNI interfaces corresponding to them. So if there >>> was eth0 and eth1 in the non-dpdk mode, once I take over the ports >>> using dpdk, I create eth0 and eth1 KNI interfaces. As far as the linux >>> network managers are concerned, they dont really know about it (or >>> care I guess) - for example the dhcp client tries getting a dhcp >>> address over these KNI interfaces and succeeds. >>> >>> Now if my application crashes, I dont want the entire network >>> management subsystem on linux and the hotplugs and this and that to >>> get alarmed and routes to vanish from the route table etc.. etc.. The >>> application will crash and come back up real quick, nothing needs to >>> change in that meantime. > > > Maybe a stupid question; why not fixing your application so that it doesn't > crash, instead of adding adhoc patches? > > marc > > >>> >>> Any way to achieve that ? I just want to keep the KNI around even if >>> my app vanishes. >>> >>> Rgds, >>> Gopa. > >
[dpdk-dev] How to prevent KNI interface from getting deleted on application termination?
Reading through the KNI module source, doesnt look like there is a way to do this. For my requirement, I will make some patch tomorrow to have a module option to just keep the KNI data structures around even if /dev/kni is closed, looks straightforward to do from the code Rgds, Gopa. On Wed, Jul 8, 2015 at 1:00 PM, Gopakumar Choorakkot Edakkunni wrote: > Hi all, > > My application takes over one/multiple ethernet port(s) in a linux > system and creates KNI interfaces corresponding to them. So if there > was eth0 and eth1 in the non-dpdk mode, once I take over the ports > using dpdk, I create eth0 and eth1 KNI interfaces. As far as the linux > network managers are concerned, they dont really know about it (or > care I guess) - for example the dhcp client tries getting a dhcp > address over these KNI interfaces and succeeds. > > Now if my application crashes, I dont want the entire network > management subsystem on linux and the hotplugs and this and that to > get alarmed and routes to vanish from the route table etc.. etc.. The > application will crash and come back up real quick, nothing needs to > change in that meantime. > > Any way to achieve that ? I just want to keep the KNI around even if > my app vanishes. > > Rgds, > Gopa.
[dpdk-dev] How to prevent KNI interface from getting deleted on application termination?
Hi all, My application takes over one/multiple ethernet port(s) in a linux system and creates KNI interfaces corresponding to them. So if there was eth0 and eth1 in the non-dpdk mode, once I take over the ports using dpdk, I create eth0 and eth1 KNI interfaces. As far as the linux network managers are concerned, they dont really know about it (or care I guess) - for example the dhcp client tries getting a dhcp address over these KNI interfaces and succeeds. Now if my application crashes, I dont want the entire network management subsystem on linux and the hotplugs and this and that to get alarmed and routes to vanish from the route table etc.. etc.. The application will crash and come back up real quick, nothing needs to change in that meantime. Any way to achieve that ? I just want to keep the KNI around even if my app vanishes. Rgds, Gopa.
[dpdk-dev] Using rte_ring_mp_xyz() across EAL and non-EAL threads ?
Hi, I have a requirement where one of my non-EAL app threads needs to handoff some packets to an EAL task. I was thinking of using rte_ring_mp_enqueue/dequeue for that purpose. I looked at the code for the rte_ring library and it doesnt look like it has any "EAL" dependencies, but I wanted to double confirm that there are no issues in using it that way. Dint find much yes/no info about that on the mailers/docs. Pls let me know your thoughts. Rgds, Gopa.
[dpdk-dev] dpdk-2.0.0: crash in ixgbe_recv_scattered_pkts_vec->_recv_raw_pkts_vec->desc_to_olflags_v
So update on this. Summary is that its purely my fault, apologies for prematurely suspecting the wrong areas. Details below 1. So my AWS box had an eth0 interface without DPDK, I enabled dpdk AND created a KNI interface also AND named the KNI interface to be eth0 2. So Ubuntu started its dhcpclient on that interface, but my app doesnt really do anything do read the dhcp (renews) from the KNI and send it out the physical port and vice versa .. The kni was just sitting there not doing much of Rx/Tx 3. Now my l2fwd-equivalent code started working fine, after a few minutes, the dhcp client on ubuntu gave up attempting dhcp renew (eth0 already had an IP) and attempted to take off the IP from eth0 4. At this point the standard KNI examples in dpdk which has callbacks registered, ended up being invoked - and the examples have a port_stop() and a port_start() in them - and exactly at this point my app crashed So my bad! I just no-oped the callbacks for now and changed AWS eht0 from dhcp to static IP and this are fine now ! My system has been up for long with no issues. Thanks again Thomas and Bruce for the quick response and suggestions Rgds, Gopa. On Tue, Jun 30, 2015 at 11:28 AM, Gopakumar Choorakkot Edakkunni wrote: > Hi Thomas, Bruce, > > Thanks for the responses. Please find my answers as below. > > Thomas>> "You mean you are using SR-IOV from Amazon, right? Do you > have more hardware details?" > > That is correct. I am attaching three files cpuinfo.txt lcpci.txt and > portconf.txt (just the port config that I am using, nothing special, > yanked off of l2fwd example). The two 82599 VF interfaces seen in > lspci output are the ones of interest - I use one of them in dpdk > mode. > > Thomas>> Did you try to disable CONFIG_RTE_IXGBE_INC_VECTOR? > > Thanks for the suggestion, I made that change and was giving it some > time. Now the result of that is not entirely black and white: > previously (in vector mode) my app used to Rx/Tx packets nicely > without any hiccups, but would crash in 10 minutes :). Now with this > suggested change, its been running for a while and doesnt crash, but > the Tx latency and Tx loss is so high (around 10% tx loss) that the > app is not doing a great job - but that might just be something that I > need to adapt to when using non-vector mode ? I will experiment on > that a bit more. So I "think" its fair to say that with the vector > disabled, theres no crash, but I need to chase this latency/loss now. > > Thomas>> Not needed. A DPDK application is fast enough to do the job > in 10 minutes ;) > > Haha, good one :). Thats where I want to get to eventually, but right > now some distance from it. > > Bruce>> Can you perhaps isolate any further the root cause of the > issue. For example, does it only occur when you get three packets at > the receive ring wraps back around to zero? > > I will try some more experiments, will read and understand this Rx > code a bit more to be able to answer the qn about whether ring wraps > around when the problem happens etc.. > > Rgds, > Gopa. > > > On Tue, Jun 30, 2015 at 9:08 AM, Thomas Monjalon > wrote: >> 2015-06-30 08:49, Gopakumar Choorakkot Edakkunni: >>> I am starting to tryout dpdk-2.0.0 with a simple Rx routine very >>> similar to the l2fwd example - I am running this on a c3.8xlarge aws >>> sr-iov enabled vpc instance (inside the vm it uses ixgbevf driver). >> >> You mean you are using SR-IOV from Amazon, right? >> Do you have more hardware details? >> >>> Once in every 10 minutes my application crashes in the recieve path. >>> And whenever I check the crash reason its because it always has three >>> packets in the burst array (I have provided array size of 32) instead >>> of the four that it tries to collect in one bunch. And inside >>> desc_to_olflags_v(), theres the assumption that there are four >>> packets, and obviously it crashes trying to access the fourth buffer. >> >> Did you try to disable CONFIG_RTE_IXGBE_INC_VECTOR? >> >>> With a brief look at the code, I really cant make out how its >>> guaranteed that we will always have four descriptors fully populated ? >>> After the first iteration, the loop does break out if (likely(var != >>> RTE_IXGBE_DESCS_PER_LOOP)), but how about the very first iteration >>> where we might not have four ? >>> >>> Any thoughts will be helpful here, trying to get my app working for >>> more than 10 minutes :) >> >> Not needed. A DPDK application is fast enough to do the job in 10 minutes ;) >>
[dpdk-dev] dpdk-2.0.0: crash in ixgbe_recv_scattered_pkts_vec->_recv_raw_pkts_vec->desc_to_olflags_v
Hi Thomas, Bruce, Thanks for the responses. Please find my answers as below. Thomas>> "You mean you are using SR-IOV from Amazon, right? Do you have more hardware details?" That is correct. I am attaching three files cpuinfo.txt lcpci.txt and portconf.txt (just the port config that I am using, nothing special, yanked off of l2fwd example). The two 82599 VF interfaces seen in lspci output are the ones of interest - I use one of them in dpdk mode. Thomas>> Did you try to disable CONFIG_RTE_IXGBE_INC_VECTOR? Thanks for the suggestion, I made that change and was giving it some time. Now the result of that is not entirely black and white: previously (in vector mode) my app used to Rx/Tx packets nicely without any hiccups, but would crash in 10 minutes :). Now with this suggested change, its been running for a while and doesnt crash, but the Tx latency and Tx loss is so high (around 10% tx loss) that the app is not doing a great job - but that might just be something that I need to adapt to when using non-vector mode ? I will experiment on that a bit more. So I "think" its fair to say that with the vector disabled, theres no crash, but I need to chase this latency/loss now. Thomas>> Not needed. A DPDK application is fast enough to do the job in 10 minutes ;) Haha, good one :). Thats where I want to get to eventually, but right now some distance from it. Bruce>> Can you perhaps isolate any further the root cause of the issue. For example, does it only occur when you get three packets at the receive ring wraps back around to zero? I will try some more experiments, will read and understand this Rx code a bit more to be able to answer the qn about whether ring wraps around when the problem happens etc.. Rgds, Gopa. On Tue, Jun 30, 2015 at 9:08 AM, Thomas Monjalon wrote: > 2015-06-30 08:49, Gopakumar Choorakkot Edakkunni: >> I am starting to tryout dpdk-2.0.0 with a simple Rx routine very >> similar to the l2fwd example - I am running this on a c3.8xlarge aws >> sr-iov enabled vpc instance (inside the vm it uses ixgbevf driver). > > You mean you are using SR-IOV from Amazon, right? > Do you have more hardware details? > >> Once in every 10 minutes my application crashes in the recieve path. >> And whenever I check the crash reason its because it always has three >> packets in the burst array (I have provided array size of 32) instead >> of the four that it tries to collect in one bunch. And inside >> desc_to_olflags_v(), theres the assumption that there are four >> packets, and obviously it crashes trying to access the fourth buffer. > > Did you try to disable CONFIG_RTE_IXGBE_INC_VECTOR? > >> With a brief look at the code, I really cant make out how its >> guaranteed that we will always have four descriptors fully populated ? >> After the first iteration, the loop does break out if (likely(var != >> RTE_IXGBE_DESCS_PER_LOOP)), but how about the very first iteration >> where we might not have four ? >> >> Any thoughts will be helpful here, trying to get my app working for >> more than 10 minutes :) > > Not needed. A DPDK application is fast enough to do the job in 10 minutes ;) > -- next part -- root at ip-10-0-0-155:/home/ubuntu# cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 62 model name : Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz stepping: 4 microcode : 0x428 cpu MHz : 2793.394 cache size : 25600 KB physical id : 0 siblings: 16 core id : 0 cpu cores : 8 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm xsaveopt fsgsbase smep erms bogomips: 5586.78 clflush size: 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management: processor : 1 vendor_id : GenuineIntel cpu family : 6 model : 62 model name : Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz stepping: 4 microcode : 0x428 cpu MHz : 2793.394 cache size : 25600 KB physical id : 0 siblings: 16 core id : 1 cpu cores : 8 apicid : 2 initial apicid : 2 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfp
[dpdk-dev] dpdk-2.0.0: crash in ixgbe_recv_scattered_pkts_vec->_recv_raw_pkts_vec->desc_to_olflags_v
Hi, I am starting to tryout dpdk-2.0.0 with a simple Rx routine very similar to the l2fwd example - I am running this on a c3.8xlarge aws sr-iov enabled vpc instance (inside the vm it uses ixgbevf driver). Once in every 10 minutes my application crashes in the recieve path. And whenever I check the crash reason its because it always has three packets in the burst array (I have provided array size of 32) instead of the four that it tries to collect in one bunch. And inside desc_to_olflags_v(), theres the assumption that there are four packets, and obviously it crashes trying to access the fourth buffer. With a brief look at the code, I really cant make out how its guaranteed that we will always have four descriptors fully populated ? After the first iteration, the loop does break out if (likely(var != RTE_IXGBE_DESCS_PER_LOOP)), but how about the very first iteration where we might not have four ? Any thoughts will be helpful here, trying to get my app working for more than 10 minutes :) #0 0x004c8c58 in desc_to_olflags_v (rx_pkts=0x7f1cb0ff17a0, descs=) at /home/gopa/dpdk-2.0.0/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c:173 #1 _recv_raw_pkts_vec (rxq=0x7f1d364fcbc0, rx_pkts=, nb_pkts=, split_packet=0x7f1cb0ff16d0 "") at /home/gopa/dpdk-2.0.0/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c:305 #2 0x004c9cea in ixgbe_recv_scattered_pkts_vec (rx_queue=0x7f1d364fcbc0, rx_pkts=0x7f1cb0ff17a0, nb_pkts=) at /home/gopa/dpdk-2.0.0/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c:475 #3 0x006b9bd8 in rte_eth_rx_burst (port_id=0 '\000', queue_id=0, rx_pkts=0x7f1cb0ff17a0, nb_pkts=32) at /home/gopa/usr/dpdk/include/rte_ethdev.h:2417 --- (gdb) frame 0 #0 0x004c8c58 in desc_to_olflags_v (rx_pkts=0x7f1cb0ff17a0, descs=) at /home/gopa/dpdk-2.0.0/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c:173 173/home/gopa/dpdk-2.0.0/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c: No such file or directory. (gdb) p rx_pkts[0] $9 = (struct rte_mbuf *) 0x7f1cf8284640 (gdb) p rx_pkts[1] $10 = (struct rte_mbuf *) 0x7f1d3991061c (gdb) p rx_pkts[2] $11 = (struct rte_mbuf *) 0x7f1d364fc740 (gdb) p rx_pkts[3] $12 = (struct rte_mbuf *) 0x0 (gdb)
