Re: [dpdk-dev] virtio "how to restart applications" - //dpdk.org/doc/virtio-net-pmd
Also Yuanhuan, your suggestion about the hugepage mapping / clearing memory was great .. I tried a test where I just wrote random values into the entire hugepage area and that succesfully crashed the ovs on the host :). So thats a good test to generally ensure that the guest doesnt mess up the host ! Thanks again for your suggestions. Rgds, Gopa. On Sat, Mar 18, 2017 at 4:43 PM, Gopakumar Choorakkot Edakkunni < gopakumar@gmail.com> wrote: > I ended up implementing a mechanism to do the equivalent of a > vtpci_reset() as soon as the dpdk-app dies and just before it comes back > up. I am "hoping" that is sufficient to let the host know that the virtio > rings etc.. are unconfigured, so that when the dpdk app comes up again in > guest and does hugepage init etc.., it in theory should not confuse the > host ovs ? > > Rgds, > Gopa. > > On Sat, Mar 18, 2017 at 2:37 PM, Gopakumar Choorakkot Edakkunni < > gopakumar@gmail.com> wrote: > >> I mean vtpci_reset is called from rte_eal_pci_probe() which is the *last* >> thing in rte_eal_init(), *after* hugepage init, so if I can somehow get >> that done *before* hugepage init maybe all will be well (because I cant do >> anything to fix the host side) >> >> Rgds, >> Gopa. >> >> On Sat, Mar 18, 2017 at 2:32 PM, Gopakumar Choorakkot Edakkunni < >> gopakumar@gmail.com> wrote: >> >>> Hi Yuan, >>> >>> As a "hack"/"workaround", in rte_eal_init(), if I can call vtpci_reset() >>> just before rte_eal_memory_init(), that should take care of the problem of >>> host zeroing out hugepages right ? As of today vtpci_reset() is called in >>> rte_eal_dev_init() which comes *after* rte_eal_memory_init() >>> >>> Rgds, >>> Gopa. >>> >>> On Thu, Mar 16, 2017 at 10:50 PM, Gopakumar Choorakkot Edakkunni < >>> gopakumar@gmail.com> wrote: >>> >>>> Thanks again Yuanhan, you are the true expert!! >>>> >>>> Rgds, >>>> Gopa. >>>> >>>> On Thu, Mar 16, 2017 at 10:40 PM, Yuanhan Liu < >>>> yuanhan@linux.intel.com> wrote: >>>> >>>>> On Thu, Mar 16, 2017 at 10:30:09PM -0700, Gopakumar Choorakkot >>>>> Edakkunni wrote: >>>>> > Thanks for the confirmation, glad I reached the person who knows the >>>>> nuts and >>>>> > bolts of virtio :-). So if the host is not in our control (ie if I >>>>> am just >>>>> > running as a VM on host provided by thirdparty vendor), is there any >>>>> workaround >>>>> > I can do from the guest side to prevent problems from happening on a >>>>> guest >>>>> > restart ? >>>>> >>>>> Not too much. You might want to hack the guest DPDK EAL memory >>>>> initiation >>>>> part though, to not reset the hugepage memory on start. But that's too >>>>> hacky >>>>> that I will not recommend you to do so! >>>>> >>>>> > And if theres no workarounds at all and the host has to change, >>>>> instead of >>>>> > asking the third party vendor to do a wholesale upgrade to 16.04, is >>>>> there one/ >>>>> > few commits that can be added to the host ovs-dpdk to take care of >>>>> this guest >>>>> > restart virtio-reset-before opening case ? >>>>> >>>>> Yes, backporting the commits I have mentioned should be able to fix it. >>>>> But please note that I did some code refactorings before those fixes: >>>>> it >>>>> won't apply cleanly to DPDK v2.2. >>>>> >>>>> And if you want to upgrade, I'd suggest to upgrade to v16.11, which is >>>>> LTS release. >>>>> >>>>> --yliu >>>>> > >>>>> > Rgds, >>>>> > Gopa. >>>>> > >>>>> > On Thu, Mar 16, 2017 at 10:24 PM, Yuanhan Liu < >>>>> yuanhan@linux.intel.com> >>>>> > wrote: >>>>> > >>>>> > On Thu, Mar 16, 2017 at 10:20:30PM -0700, Gopakumar Choorakkot >>>>> Edakkunni >>>>> > wrote: >>>>> > > >> When I was saying dpdk version, I meant the DPDK version >>>>> with OVS. >>>>> > > >>>>> > > Oh I see! My apologies for
Re: [dpdk-dev] virtio "how to restart applications" - //dpdk.org/doc/virtio-net-pmd
I ended up implementing a mechanism to do the equivalent of a vtpci_reset() as soon as the dpdk-app dies and just before it comes back up. I am "hoping" that is sufficient to let the host know that the virtio rings etc.. are unconfigured, so that when the dpdk app comes up again in guest and does hugepage init etc.., it in theory should not confuse the host ovs ? Rgds, Gopa. On Sat, Mar 18, 2017 at 2:37 PM, Gopakumar Choorakkot Edakkunni < gopakumar@gmail.com> wrote: > I mean vtpci_reset is called from rte_eal_pci_probe() which is the *last* > thing in rte_eal_init(), *after* hugepage init, so if I can somehow get > that done *before* hugepage init maybe all will be well (because I cant do > anything to fix the host side) > > Rgds, > Gopa. > > On Sat, Mar 18, 2017 at 2:32 PM, Gopakumar Choorakkot Edakkunni < > gopakumar@gmail.com> wrote: > >> Hi Yuan, >> >> As a "hack"/"workaround", in rte_eal_init(), if I can call vtpci_reset() >> just before rte_eal_memory_init(), that should take care of the problem of >> host zeroing out hugepages right ? As of today vtpci_reset() is called in >> rte_eal_dev_init() which comes *after* rte_eal_memory_init() >> >> Rgds, >> Gopa. >> >> On Thu, Mar 16, 2017 at 10:50 PM, Gopakumar Choorakkot Edakkunni < >> gopakumar@gmail.com> wrote: >> >>> Thanks again Yuanhan, you are the true expert!! >>> >>> Rgds, >>> Gopa. >>> >>> On Thu, Mar 16, 2017 at 10:40 PM, Yuanhan Liu < >>> yuanhan@linux.intel.com> wrote: >>> >>>> On Thu, Mar 16, 2017 at 10:30:09PM -0700, Gopakumar Choorakkot >>>> Edakkunni wrote: >>>> > Thanks for the confirmation, glad I reached the person who knows the >>>> nuts and >>>> > bolts of virtio :-). So if the host is not in our control (ie if I am >>>> just >>>> > running as a VM on host provided by thirdparty vendor), is there any >>>> workaround >>>> > I can do from the guest side to prevent problems from happening on a >>>> guest >>>> > restart ? >>>> >>>> Not too much. You might want to hack the guest DPDK EAL memory >>>> initiation >>>> part though, to not reset the hugepage memory on start. But that's too >>>> hacky >>>> that I will not recommend you to do so! >>>> >>>> > And if theres no workarounds at all and the host has to change, >>>> instead of >>>> > asking the third party vendor to do a wholesale upgrade to 16.04, is >>>> there one/ >>>> > few commits that can be added to the host ovs-dpdk to take care of >>>> this guest >>>> > restart virtio-reset-before opening case ? >>>> >>>> Yes, backporting the commits I have mentioned should be able to fix it. >>>> But please note that I did some code refactorings before those fixes: it >>>> won't apply cleanly to DPDK v2.2. >>>> >>>> And if you want to upgrade, I'd suggest to upgrade to v16.11, which is >>>> LTS release. >>>> >>>> --yliu >>>> > >>>> > Rgds, >>>> > Gopa. >>>> > >>>> > On Thu, Mar 16, 2017 at 10:24 PM, Yuanhan Liu < >>>> yuanhan@linux.intel.com> >>>> > wrote: >>>> > >>>> > On Thu, Mar 16, 2017 at 10:20:30PM -0700, Gopakumar Choorakkot >>>> Edakkunni >>>> > wrote: >>>> > > >> When I was saying dpdk version, I meant the DPDK version >>>> with OVS. >>>> > > >>>> > > Oh I see! My apologies for the misuderstanding. The dpdk >>>> version used by >>>> > host >>>> > > ovs should be dpdk2.2, the guest process uses dpdk16.07. The >>>> OVS process >>>> > is not >>>> > > getting restarted, what is getting restarted is the guest >>>> process using >>>> > > dpdk16.07 - so the above clarifications you had about virtio >>>> being >>>> > > reset-before-opened on guest restart - does that still hold >>>> good or does >>>> > that >>>> > > need the HOST side dpdk to be 16.04 or above ? >>>> > >>>> > Yes, the HOST dpdk should be >= v16.04. >>>> > >>>> > -
Re: [dpdk-dev] virtio "how to restart applications" - //dpdk.org/doc/virtio-net-pmd
I mean vtpci_reset is called from rte_eal_pci_probe() which is the *last* thing in rte_eal_init(), *after* hugepage init, so if I can somehow get that done *before* hugepage init maybe all will be well (because I cant do anything to fix the host side) Rgds, Gopa. On Sat, Mar 18, 2017 at 2:32 PM, Gopakumar Choorakkot Edakkunni < gopakumar@gmail.com> wrote: > Hi Yuan, > > As a "hack"/"workaround", in rte_eal_init(), if I can call vtpci_reset() > just before rte_eal_memory_init(), that should take care of the problem of > host zeroing out hugepages right ? As of today vtpci_reset() is called in > rte_eal_dev_init() which comes *after* rte_eal_memory_init() > > Rgds, > Gopa. > > On Thu, Mar 16, 2017 at 10:50 PM, Gopakumar Choorakkot Edakkunni < > gopakumar@gmail.com> wrote: > >> Thanks again Yuanhan, you are the true expert!! >> >> Rgds, >> Gopa. >> >> On Thu, Mar 16, 2017 at 10:40 PM, Yuanhan Liu < >> yuanhan@linux.intel.com> wrote: >> >>> On Thu, Mar 16, 2017 at 10:30:09PM -0700, Gopakumar Choorakkot Edakkunni >>> wrote: >>> > Thanks for the confirmation, glad I reached the person who knows the >>> nuts and >>> > bolts of virtio :-). So if the host is not in our control (ie if I am >>> just >>> > running as a VM on host provided by thirdparty vendor), is there any >>> workaround >>> > I can do from the guest side to prevent problems from happening on a >>> guest >>> > restart ? >>> >>> Not too much. You might want to hack the guest DPDK EAL memory initiation >>> part though, to not reset the hugepage memory on start. But that's too >>> hacky >>> that I will not recommend you to do so! >>> >>> > And if theres no workarounds at all and the host has to change, >>> instead of >>> > asking the third party vendor to do a wholesale upgrade to 16.04, is >>> there one/ >>> > few commits that can be added to the host ovs-dpdk to take care of >>> this guest >>> > restart virtio-reset-before opening case ? >>> >>> Yes, backporting the commits I have mentioned should be able to fix it. >>> But please note that I did some code refactorings before those fixes: it >>> won't apply cleanly to DPDK v2.2. >>> >>> And if you want to upgrade, I'd suggest to upgrade to v16.11, which is >>> LTS release. >>> >>> --yliu >>> > >>> > Rgds, >>> > Gopa. >>> > >>> > On Thu, Mar 16, 2017 at 10:24 PM, Yuanhan Liu < >>> yuanhan@linux.intel.com> >>> > wrote: >>> > >>> > On Thu, Mar 16, 2017 at 10:20:30PM -0700, Gopakumar Choorakkot >>> Edakkunni >>> > wrote: >>> > > >> When I was saying dpdk version, I meant the DPDK version with >>> OVS. >>> > > >>> > > Oh I see! My apologies for the misuderstanding. The dpdk version >>> used by >>> > host >>> > > ovs should be dpdk2.2, the guest process uses dpdk16.07. The OVS >>> process >>> > is not >>> > > getting restarted, what is getting restarted is the guest >>> process using >>> > > dpdk16.07 - so the above clarifications you had about virtio >>> being >>> > > reset-before-opened on guest restart - does that still hold good >>> or does >>> > that >>> > > need the HOST side dpdk to be 16.04 or above ? >>> > >>> > Yes, the HOST dpdk should be >= v16.04. >>> > >>> > --yliu >>> > > >>> > > >> And yes, the fixes are not included in the DPDK required for >>> OVS 2.4. >>> > > >>> > > Thanks for the info. >>> > > >>> > > Rgds, >>> > > Gopa. >>> > > >>> > > On Thu, Mar 16, 2017 at 10:13 PM, Yuanhan Liu < >>> > yuanhan@linux.intel.com> >>> > > wrote: >>> > > >>> > > On Thu, Mar 16, 2017 at 09:56:01PM -0700, Gopakumar >>> Choorakkot >>> > Edakkunni >>> > > wrote: >>> > > > Hi Yuanhan, >>> > > > >>> > > > Thanks for the confirmation about not having to do >>> anythi
Re: [dpdk-dev] virtio "how to restart applications" - //dpdk.org/doc/virtio-net-pmd
Hi Yuan, As a "hack"/"workaround", in rte_eal_init(), if I can call vtpci_reset() just before rte_eal_memory_init(), that should take care of the problem of host zeroing out hugepages right ? As of today vtpci_reset() is called in rte_eal_dev_init() which comes *after* rte_eal_memory_init() Rgds, Gopa. On Thu, Mar 16, 2017 at 10:50 PM, Gopakumar Choorakkot Edakkunni < gopakumar@gmail.com> wrote: > Thanks again Yuanhan, you are the true expert!! > > Rgds, > Gopa. > > On Thu, Mar 16, 2017 at 10:40 PM, Yuanhan Liu > wrote: > >> On Thu, Mar 16, 2017 at 10:30:09PM -0700, Gopakumar Choorakkot Edakkunni >> wrote: >> > Thanks for the confirmation, glad I reached the person who knows the >> nuts and >> > bolts of virtio :-). So if the host is not in our control (ie if I am >> just >> > running as a VM on host provided by thirdparty vendor), is there any >> workaround >> > I can do from the guest side to prevent problems from happening on a >> guest >> > restart ? >> >> Not too much. You might want to hack the guest DPDK EAL memory initiation >> part though, to not reset the hugepage memory on start. But that's too >> hacky >> that I will not recommend you to do so! >> >> > And if theres no workarounds at all and the host has to change, instead >> of >> > asking the third party vendor to do a wholesale upgrade to 16.04, is >> there one/ >> > few commits that can be added to the host ovs-dpdk to take care of this >> guest >> > restart virtio-reset-before opening case ? >> >> Yes, backporting the commits I have mentioned should be able to fix it. >> But please note that I did some code refactorings before those fixes: it >> won't apply cleanly to DPDK v2.2. >> >> And if you want to upgrade, I'd suggest to upgrade to v16.11, which is >> LTS release. >> >> --yliu >> > >> > Rgds, >> > Gopa. >> > >> > On Thu, Mar 16, 2017 at 10:24 PM, Yuanhan Liu < >> yuanhan@linux.intel.com> >> > wrote: >> > >> > On Thu, Mar 16, 2017 at 10:20:30PM -0700, Gopakumar Choorakkot >> Edakkunni >> > wrote: >> > > >> When I was saying dpdk version, I meant the DPDK version with >> OVS. >> > > >> > > Oh I see! My apologies for the misuderstanding. The dpdk version >> used by >> > host >> > > ovs should be dpdk2.2, the guest process uses dpdk16.07. The OVS >> process >> > is not >> > > getting restarted, what is getting restarted is the guest process >> using >> > > dpdk16.07 - so the above clarifications you had about virtio being >> > > reset-before-opened on guest restart - does that still hold good >> or does >> > that >> > > need the HOST side dpdk to be 16.04 or above ? >> > >> > Yes, the HOST dpdk should be >= v16.04. >> > >> > --yliu >> > > >> > > >> And yes, the fixes are not included in the DPDK required for >> OVS 2.4. >> > > >> > > Thanks for the info. >> > > >> > > Rgds, >> > > Gopa. >> > > >> > > On Thu, Mar 16, 2017 at 10:13 PM, Yuanhan Liu < >> > yuanhan@linux.intel.com> >> > > wrote: >> > > >> > > On Thu, Mar 16, 2017 at 09:56:01PM -0700, Gopakumar Choorakkot >> > Edakkunni >> > > wrote: >> > > > Hi Yuanhan, >> > > > >> > > > Thanks for the confirmation about not having to do anything >> special >> > to >> > > close >> > > > the ports on dpdk going down or coming up. >> > > > >> > > > As for the question about if I met any issue of ovs getting >> stuck - >> > yes, >> > > my >> > > > guest process runs dpdk 16.07 as I mentioned earlier - and >> if I >> > kill my >> > > guest >> > > > process, then the host OVS-dpdk on the host reports stall ! >> The >> > OVS-dpdk >> > > and >> > > > emu versions I use are as below. But maybe that is because >> of the >> > ovs >> > > missing >> > > > th
Re: [dpdk-dev] virtio "how to restart applications" - //dpdk.org/doc/virtio-net-pmd
Thanks again Yuanhan, you are the true expert!! Rgds, Gopa. On Thu, Mar 16, 2017 at 10:40 PM, Yuanhan Liu wrote: > On Thu, Mar 16, 2017 at 10:30:09PM -0700, Gopakumar Choorakkot Edakkunni > wrote: > > Thanks for the confirmation, glad I reached the person who knows the > nuts and > > bolts of virtio :-). So if the host is not in our control (ie if I am > just > > running as a VM on host provided by thirdparty vendor), is there any > workaround > > I can do from the guest side to prevent problems from happening on a > guest > > restart ? > > Not too much. You might want to hack the guest DPDK EAL memory initiation > part though, to not reset the hugepage memory on start. But that's too > hacky > that I will not recommend you to do so! > > > And if theres no workarounds at all and the host has to change, instead > of > > asking the third party vendor to do a wholesale upgrade to 16.04, is > there one/ > > few commits that can be added to the host ovs-dpdk to take care of this > guest > > restart virtio-reset-before opening case ? > > Yes, backporting the commits I have mentioned should be able to fix it. > But please note that I did some code refactorings before those fixes: it > won't apply cleanly to DPDK v2.2. > > And if you want to upgrade, I'd suggest to upgrade to v16.11, which is > LTS release. > > --yliu > > > > Rgds, > > Gopa. > > > > On Thu, Mar 16, 2017 at 10:24 PM, Yuanhan Liu < > yuanhan@linux.intel.com> > > wrote: > > > > On Thu, Mar 16, 2017 at 10:20:30PM -0700, Gopakumar Choorakkot > Edakkunni > > wrote: > > > >> When I was saying dpdk version, I meant the DPDK version with > OVS. > > > > > > Oh I see! My apologies for the misuderstanding. The dpdk version > used by > > host > > > ovs should be dpdk2.2, the guest process uses dpdk16.07. The OVS > process > > is not > > > getting restarted, what is getting restarted is the guest process > using > > > dpdk16.07 - so the above clarifications you had about virtio being > > > reset-before-opened on guest restart - does that still hold good > or does > > that > > > need the HOST side dpdk to be 16.04 or above ? > > > > Yes, the HOST dpdk should be >= v16.04. > > > > --yliu > > > > > > >> And yes, the fixes are not included in the DPDK required for > OVS 2.4. > > > > > > Thanks for the info. > > > > > > Rgds, > > > Gopa. > > > > > > On Thu, Mar 16, 2017 at 10:13 PM, Yuanhan Liu < > > yuanhan@linux.intel.com> > > > wrote: > > > > > > On Thu, Mar 16, 2017 at 09:56:01PM -0700, Gopakumar Choorakkot > > Edakkunni > > > wrote: > > > > Hi Yuanhan, > > > > > > > > Thanks for the confirmation about not having to do anything > special > > to > > > close > > > > the ports on dpdk going down or coming up. > > > > > > > > As for the question about if I met any issue of ovs getting > stuck - > > yes, > > > my > > > > guest process runs dpdk 16.07 as I mentioned earlier - and > if I > > kill my > > > guest > > > > process, then the host OVS-dpdk on the host reports stall ! > The > > OVS-dpdk > > > and > > > > emu versions I use are as below. But maybe that is because > of the > > ovs > > > missing > > > > the fixes you mentioned ? > > > > > > When I was saying dpdk version, I meant the DPDK version with > OVS. > > > > > > > ~# ovs-vswitchd --version > > > > ovs-vswitchd (Open vSwitch) 2.4.1 > > > > > > And yes, the fixes are not included in the DPDK required for > OVS 2.4. > > > > > > --yliu > > > > > > > Compiled Nov 14 2016 06:53:31 > > > > # kvm --version > > > > QEMU emulator version 2.2.0, Copyright (c) 2003-2008 Fabrice > > Bellard > > > > ~# > > > > > > > > > > > > Rgds, > > > > Gopa. > > > > > > > > On Th
Re: [dpdk-dev] virtio "how to restart applications" - //dpdk.org/doc/virtio-net-pmd
Thanks for the confirmation, glad I reached the person who knows the nuts and bolts of virtio :-). So if the host is not in our control (ie if I am just running as a VM on host provided by thirdparty vendor), is there any workaround I can do from the guest side to prevent problems from happening on a guest restart ? And if theres no workarounds at all and the host has to change, instead of asking the third party vendor to do a wholesale upgrade to 16.04, is there one/few commits that can be added to the host ovs-dpdk to take care of this guest restart virtio-reset-before opening case ? Rgds, Gopa. On Thu, Mar 16, 2017 at 10:24 PM, Yuanhan Liu wrote: > On Thu, Mar 16, 2017 at 10:20:30PM -0700, Gopakumar Choorakkot Edakkunni > wrote: > > >> When I was saying dpdk version, I meant the DPDK version with OVS. > > > > Oh I see! My apologies for the misuderstanding. The dpdk version used by > host > > ovs should be dpdk2.2, the guest process uses dpdk16.07. The OVS process > is not > > getting restarted, what is getting restarted is the guest process using > > dpdk16.07 - so the above clarifications you had about virtio being > > reset-before-opened on guest restart - does that still hold good or does > that > > need the HOST side dpdk to be 16.04 or above ? > > Yes, the HOST dpdk should be >= v16.04. > > --yliu > > > > >> And yes, the fixes are not included in the DPDK required for OVS 2.4. > > > > Thanks for the info. > > > > Rgds, > > Gopa. > > > > On Thu, Mar 16, 2017 at 10:13 PM, Yuanhan Liu < > yuanhan@linux.intel.com> > > wrote: > > > > On Thu, Mar 16, 2017 at 09:56:01PM -0700, Gopakumar Choorakkot > Edakkunni > > wrote: > > > Hi Yuanhan, > > > > > > Thanks for the confirmation about not having to do anything > special to > > close > > > the ports on dpdk going down or coming up. > > > > > > As for the question about if I met any issue of ovs getting stuck > - yes, > > my > > > guest process runs dpdk 16.07 as I mentioned earlier - and if I > kill my > > guest > > > process, then the host OVS-dpdk on the host reports stall ! The > OVS-dpdk > > and > > > emu versions I use are as below. But maybe that is because of the > ovs > > missing > > > the fixes you mentioned ? > > > > When I was saying dpdk version, I meant the DPDK version with OVS. > > > > > ~# ovs-vswitchd --version > > > ovs-vswitchd (Open vSwitch) 2.4.1 > > > > And yes, the fixes are not included in the DPDK required for OVS 2.4. > > > > --yliu > > > > > Compiled Nov 14 2016 06:53:31 > > > # kvm --version > > > QEMU emulator version 2.2.0, Copyright (c) 2003-2008 Fabrice > Bellard > > > ~# > > > > > > > > > Rgds, > > > Gopa. > > > > > > On Thu, Mar 16, 2017 at 9:35 PM, Yuanhan Liu < > yuanhan@linux.intel.com > > > > > > wrote: > > > > > > On Thu, Mar 16, 2017 at 07:48:28PM -0700, Gopakumar Choorakkot > > Edakkunni > > > wrote: > > > > Thanks a lot for the response Yuanhan. I am using dpdk > v16.07. So > > what > > > you are > > > > saying is that in 16.07, we dont really need to call > > rte_eth_dev_close() > > > on > > > > exit, > > > > > > It's not about "don't really need", it's more like "it's hard > to". > > Just > > > think that it may crash at any time. > > > > > > > because dpdk will ensure that it will do virtio reset before > init > > when it > > > > comes up right ? > > > > > > No, It just handles the abnormal case well when guest APP > restarts. > > > > > > > Regarding the vhost commits you mentioned - do we still need > those > > fixes > > > if we > > > > have the "virtio reset before init" mechanism ? > > > > > > Yes, we still need them: just think some malicious guest may > also > > forge > > > data like that. > > > > > > I'm a bit confused then. Have you actually met any issue (like > got > > stucked) > >
Re: [dpdk-dev] virtio "how to restart applications" - //dpdk.org/doc/virtio-net-pmd
>> When I was saying dpdk version, I meant the DPDK version with OVS. Oh I see! My apologies for the misuderstanding. The dpdk version used by host ovs should be dpdk2.2, the guest process uses dpdk16.07. The OVS process is not getting restarted, what is getting restarted is the guest process using dpdk16.07 - so the above clarifications you had about virtio being reset-before-opened on guest restart - does that still hold good or does that need the HOST side dpdk to be 16.04 or above ? >> And yes, the fixes are not included in the DPDK required for OVS 2.4. Thanks for the info. Rgds, Gopa. On Thu, Mar 16, 2017 at 10:13 PM, Yuanhan Liu wrote: > On Thu, Mar 16, 2017 at 09:56:01PM -0700, Gopakumar Choorakkot Edakkunni > wrote: > > Hi Yuanhan, > > > > Thanks for the confirmation about not having to do anything special to > close > > the ports on dpdk going down or coming up. > > > > As for the question about if I met any issue of ovs getting stuck - yes, > my > > guest process runs dpdk 16.07 as I mentioned earlier - and if I kill my > guest > > process, then the host OVS-dpdk on the host reports stall ! The OVS-dpdk > and > > emu versions I use are as below. But maybe that is because of the ovs > missing > > the fixes you mentioned ? > > When I was saying dpdk version, I meant the DPDK version with OVS. > > > ~# ovs-vswitchd --version > > ovs-vswitchd (Open vSwitch) 2.4.1 > > And yes, the fixes are not included in the DPDK required for OVS 2.4. > > --yliu > > > Compiled Nov 14 2016 06:53:31 > > # kvm --version > > QEMU emulator version 2.2.0, Copyright (c) 2003-2008 Fabrice Bellard > > ~# > > > > > > Rgds, > > Gopa. > > > > On Thu, Mar 16, 2017 at 9:35 PM, Yuanhan Liu < > yuanhan@linux.intel.com> > > wrote: > > > > On Thu, Mar 16, 2017 at 07:48:28PM -0700, Gopakumar Choorakkot > Edakkunni > > wrote: > > > Thanks a lot for the response Yuanhan. I am using dpdk v16.07. So > what > > you are > > > saying is that in 16.07, we dont really need to call > rte_eth_dev_close() > > on > > > exit, > > > > It's not about "don't really need", it's more like "it's hard to". > Just > > think that it may crash at any time. > > > > > because dpdk will ensure that it will do virtio reset before init > when it > > > comes up right ? > > > > No, It just handles the abnormal case well when guest APP restarts. > > > > > Regarding the vhost commits you mentioned - do we still need those > fixes > > if we > > > have the "virtio reset before init" mechanism ? > > > > Yes, we still need them: just think some malicious guest may also > forge > > data like that. > > > > I'm a bit confused then. Have you actually met any issue (like got > stucked) > > with DPDK v16.07? > > > > --yliu > > > > > Or that is a seperate problem > > > altogether (and hence we would need those fixes) ? > > > > > > Rgds, > > > Gopa. > > > > > > On Thu, Mar 16, 2017 at 7:06 PM, Yuanhan Liu < > yuanhan@linux.intel.com > > > > > > wrote: > > > > > > On Thu, Mar 16, 2017 at 12:39:16PM -0700, Gopakumar Choorakkot > > Edakkunni > > > wrote: > > > > So the doc says we should call rte_eth_dev_close() *before* > going > > down. > > > And I > > > > know that especially in dpdk-virtionet in the guest + > ovs-dpdk in > > the > > > host, > > > > the ovs ends up getting stalled/stuck (!!) if I dont close > the port > > > before > > > > starting() it when the guest dpdk process comes back up. > > > > > > I'm assuming you were using an old version, something like > dpdk v2.2? > > > IIRC, DPDK v16.04 should have fixed your issue. > > > > > > > Considering that this not done properly can screw up the > HOST ovs, > > and I > > > want > > > > to do everything possible to avoid that, I want to be 200% > sure > > that I > > > call > > > > close even if my process gets a kill -9 .. So obviously the > only > > way of > > > doing > > > &g
Re: [dpdk-dev] virtio "how to restart applications" - //dpdk.org/doc/virtio-net-pmd
Hi Yuanhan, Thanks for the confirmation about not having to do anything special to close the ports on dpdk going down or coming up. As for the question about if I met any issue of ovs getting stuck - yes, my guest process runs dpdk 16.07 as I mentioned earlier - and if I kill my guest process, then the host OVS-dpdk on the host reports stall ! The OVS-dpdk and emu versions I use are as below. But maybe that is because of the ovs missing the fixes you mentioned ? ~# ovs-vswitchd --version ovs-vswitchd (Open vSwitch) 2.4.1 Compiled Nov 14 2016 06:53:31 # kvm --version QEMU emulator version 2.2.0, Copyright (c) 2003-2008 Fabrice Bellard ~# Rgds, Gopa. On Thu, Mar 16, 2017 at 9:35 PM, Yuanhan Liu wrote: > On Thu, Mar 16, 2017 at 07:48:28PM -0700, Gopakumar Choorakkot Edakkunni > wrote: > > Thanks a lot for the response Yuanhan. I am using dpdk v16.07. So what > you are > > saying is that in 16.07, we dont really need to call rte_eth_dev_close() > on > > exit, > > It's not about "don't really need", it's more like "it's hard to". Just > think that it may crash at any time. > > > because dpdk will ensure that it will do virtio reset before init when it > > comes up right ? > > No, It just handles the abnormal case well when guest APP restarts. > > > Regarding the vhost commits you mentioned - do we still need those fixes > if we > > have the "virtio reset before init" mechanism ? > > Yes, we still need them: just think some malicious guest may also forge > data like that. > > I'm a bit confused then. Have you actually met any issue (like got stucked) > with DPDK v16.07? > > --yliu > > > Or that is a seperate problem > > altogether (and hence we would need those fixes) ? > > > > Rgds, > > Gopa. > > > > On Thu, Mar 16, 2017 at 7:06 PM, Yuanhan Liu < > yuanhan@linux.intel.com> > > wrote: > > > > On Thu, Mar 16, 2017 at 12:39:16PM -0700, Gopakumar Choorakkot > Edakkunni > > wrote: > > > So the doc says we should call rte_eth_dev_close() *before* going > down. > > And I > > > know that especially in dpdk-virtionet in the guest + ovs-dpdk in > the > > host, > > > the ovs ends up getting stalled/stuck (!!) if I dont close the port > > before > > > starting() it when the guest dpdk process comes back up. > > > > I'm assuming you were using an old version, something like dpdk v2.2? > > IIRC, DPDK v16.04 should have fixed your issue. > > > > > Considering that this not done properly can screw up the HOST ovs, > and I > > want > > > to do everything possible to avoid that, I want to be 200% sure > that I > > call > > > close even if my process gets a kill -9 .. So obviously the only > way of > > doing > > > that is to close the port when the dpdk process comes back up and > > *before* we > > > init the port. rte_eth_dev_close() is not capable of doing that as > it > > expects > > > the port parameters to be initialized etc.. before it can be > called. > > > > We do virtio reset before init, which is basically what > rte_eth_dev_close() > > mainly does. So I see no big issue here. > > > > The stuck issue is due to hugepage reset by the guest DPDK > application, > > leading all virtio vring elements being mem zeroed. The old vhost > doesn't > > handle it well, as a result, it got stuck. And here are some relevant > > commits: > > > > a436f53 vhost: avoid dead loop chain > > c687b0b vhost: check for ring descriptors overflow > > 623bc47 vhost: do sanity check for ring descriptor length > > > > --yliu > > > > > Any other > > > suggestions on what can be done to close on restart rather than > close on > > going > > > down ? Thought of bouncing this by the alias before I add a > version of > > close > > > myself that can do this close-on-restart > > > > >
Re: [dpdk-dev] virtio "how to restart applications" - //dpdk.org/doc/virtio-net-pmd
Thanks a lot for the response Yuanhan. I am using dpdk v16.07. So what you are saying is that in 16.07, we dont really need to call rte_eth_dev_close() on exit, because dpdk will ensure that it will do virtio reset before init when it comes up right ? Regarding the vhost commits you mentioned - do we still need those fixes if we have the "virtio reset before init" mechanism ? Or that is a seperate problem altogether (and hence we would need those fixes) ? Rgds, Gopa. On Thu, Mar 16, 2017 at 7:06 PM, Yuanhan Liu wrote: > On Thu, Mar 16, 2017 at 12:39:16PM -0700, Gopakumar Choorakkot Edakkunni > wrote: > > So the doc says we should call rte_eth_dev_close() *before* going down. > And I > > know that especially in dpdk-virtionet in the guest + ovs-dpdk in the > host, > > the ovs ends up getting stalled/stuck (!!) if I dont close the port > before > > starting() it when the guest dpdk process comes back up. > > I'm assuming you were using an old version, something like dpdk v2.2? > IIRC, DPDK v16.04 should have fixed your issue. > > > Considering that this not done properly can screw up the HOST ovs, and I > want > > to do everything possible to avoid that, I want to be 200% sure that I > call > > close even if my process gets a kill -9 .. So obviously the only way of > doing > > that is to close the port when the dpdk process comes back up and > *before* we > > init the port. rte_eth_dev_close() is not capable of doing that as it > expects > > the port parameters to be initialized etc.. before it can be called. > > We do virtio reset before init, which is basically what rte_eth_dev_close() > mainly does. So I see no big issue here. > > The stuck issue is due to hugepage reset by the guest DPDK application, > leading all virtio vring elements being mem zeroed. The old vhost doesn't > handle it well, as a result, it got stuck. And here are some relevant > commits: > > a436f53 vhost: avoid dead loop chain > c687b0b vhost: check for ring descriptors overflow > 623bc47 vhost: do sanity check for ring descriptor length > > --yliu > > > Any other > > suggestions on what can be done to close on restart rather than close on > going > > down ? Thought of bouncing this by the alias before I add a version of > close > > myself that can do this close-on-restart >
[dpdk-dev] virtio "how to restart applications" - //dpdk.org/doc/virtio-net-pmd
So the doc says we should call rte_eth_dev_close() *before* going down. And I know that especially in dpdk-virtionet in the guest + ovs-dpdk in the host, the ovs ends up getting stalled/stuck (!!) if I dont close the port before starting() it when the guest dpdk process comes back up. Considering that this not done properly can screw up the HOST ovs, and I want to do everything possible to avoid that, I want to be 200% sure that I call close even if my process gets a kill -9 .. So obviously the only way of doing that is to close the port when the dpdk process comes back up and *before* we init the port. rte_eth_dev_close() is not capable of doing that as it expects the port parameters to be initialized etc.. before it can be called. Any other suggestions on what can be done to close on restart rather than close on going down ? Thought of bouncing this by the alias before I add a version of close myself that can do this close-on-restart Rgds, Gopa.
Re: [dpdk-dev] Bug in virtqueue_dequeue_burst_rx()
Hi Yuanahn,
Thanks for the response. Because of my filters, I missed this email
completely, apologies for the late response ! I haven't tried breaking out
of the loop to see if it will go back to working state - I have two
testbeds, one using plain linux kvm + guest VM running 16.07 dpdk where I
see the problem. I have another testved where I use ovs-dpdk + the same
guest VM image running 16.07 dpdk. The issue happens only on the linux kvm
setup, so for now I just switched to using ovs-dpdk instead. I will go back
sometime to the linux kvm setup and try breaking of the loop and see what
happens.
Without hardly any knowledge of virtio myself, I am guessing this issue is
because the "host" said it has put one buffer into the virtio ring whereas
it really did not put the buffer, right ? So is the issue likely to be in
the host virtio layer rather than in guest dpdk ?
As for the linux-kvm setup, the trigger is very straightforward. Its just
sending traffic. The details of my setup are as below. If you have a
similar setup, I can possibly try and help you recreate the issue or maybe
even provide access to my testbed if you would like that.
1. The guest VM runs dpdk 16.07 and has 8 virtio interfaces mapping to 8
different bridges on the host. The VM has 8Gig memory with 1Gig hugepage
size and 1 hugepage (1gig memory) given to dpdk. And the VM is backed up by
16 x 1Gig hugepages on the host. I am attaching the guest XML file here
2. The host runs the below version. And all that I do is send iperf traffic
that comes in on one of the 8 interfaces in the guest VM and goes out of
another interface in the guest VM.
root@ubuntu:/home/vcadmin# uname -a
Linux ubuntu 3.16.0-77-generic #99~14.04.1-Ubuntu SMP Tue Jun 28 19:17:10
UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
root@ubuntu:/home/vcadmin# lsb_release -a
No LSB modules are available.
Distributor ID:Ubuntu
Description:Ubuntu 14.04.5 LTS
Release:14.04
Codename:trusty
root@ubuntu:/home/vcadmin# cat /proc/meminfo | grep Huge
AnonHugePages:141312 kB
HugePages_Total: 16
HugePages_Free: 16
HugePages_Rsvd:0
HugePages_Surp:0
Hugepagesize:1048576 kB
Rgds,
Gopa.
On Wed, Dec 21, 2016 at 11:38 PM, Yuanhan Liu
wrote:
> On Mon, Dec 19, 2016 at 09:59:33PM -0800, Gopakumar Choorakkot Edakkunni
> wrote:
> > While I was testing virtio with ubuntu 14.04 kvm as host and dpdk16.07
> > linux as guest, quite often I have seen that I get into a situation where
> > virtio_recv_mergeable_pkts() gets into a forever loop, after sending
> > traffic for a while. In the below API, I see that it clearly leads to a
> > while loop, I am not quite familiar with virtio or mergeable buffers, so
> > thought of checking with dpdk alias on the intent here.
> >
> > I checked the linux kernel virtio_net.c file which does the similar
> > mergeable recieve, and the kernel code breaks out in case of error.
> > Shouldnt dpdk be breaking out of here on error instead of continue ?
>
> Yep, that looks buggy.
>
> >
> > virtio_recv_mergeable_pkts()
> > {
> >
> > while (i < nb_used) {
> >
> > *num = virtqueue_dequeue_burst_rx(rxvq, rcv_pkts, len, 1);*
> > *if (num != 1)*
> > *continue;*
>
> However, normally, virtqueue_dequeue_burst_rx() would be successful
> here: the outer check (i < nb_used) somehow ascertains it.
>
> Two options I can think of so far:
>
> - will it go back working once you do "break" here?
>
> - tell us how you managed to trigger this issue, so that I could
> also reproduce it that I can debug it.
>
> --yliu
>
[dpdk-dev] Bug in virtqueue_dequeue_burst_rx()
While I was testing virtio with ubuntu 14.04 kvm as host and dpdk16.07
linux as guest, quite often I have seen that I get into a situation where
virtio_recv_mergeable_pkts() gets into a forever loop, after sending
traffic for a while. In the below API, I see that it clearly leads to a
while loop, I am not quite familiar with virtio or mergeable buffers, so
thought of checking with dpdk alias on the intent here.
I checked the linux kernel virtio_net.c file which does the similar
mergeable recieve, and the kernel code breaks out in case of error.
Shouldnt dpdk be breaking out of here on error instead of continue ?
virtio_recv_mergeable_pkts()
{
while (i < nb_used) {
*num = virtqueue_dequeue_burst_rx(rxvq, rcv_pkts, len, 1);*
*if (num != 1)*
*continue;*
}
Re: [dpdk-dev] dpdk16.07 and virtio "Cannot mmap IO port resource"
nevermind .. I saw the hew ARCH_X86 flag which I dint realise I have to add
in my config file when I moved from dpdk2.0
On Sun, Dec 11, 2016 at 12:29 AM, Gopakumar Choorakkot Edakkunni <
gopakumar@gmail.com> wrote:
> Somewhere I read that the "offset" parameter for sysfs mmap() needs to be
> the start address of the resource, so I tried the below diff, and mmap
> still fails .. Still scratching my head
>
> diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
> b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
> index 1786b75..e882940 100644
> --- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
> +++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
> @@ -462,8 +462,8 @@ pci_uio_ioport_map(struct rte_pci_device *dev, int bar,
> strerror(errno));
> goto error;
> }
> - addr = mmap(NULL, end_addr + 1, PROT_READ | PROT_WRITE,
> - MAP_SHARED, fd, 0);
> + addr = mmap(NULL, (end_addr-phys_addr) + 1, PROT_READ | PROT_WRITE,
> + MAP_SHARED, fd, phys_addr);
> close(fd);
> if (addr == MAP_FAILED) {
> RTE_LOG(ERR, EAL, "Cannot mmap IO port resource: %s\n",
> @@ -473,7 +473,7 @@ pci_uio_ioport_map(struct rte_pci_device *dev, int bar,
>
> /* strangely, the base address is mmap addr + phys_addr */
> p->base = (uintptr_t)addr + phys_addr;
> - p->len = end_addr + 1;
> + p->len = (end_addr-phys_addr) + 1;
> RTE_LOG(DEBUG, EAL, "PCI Port IO found start=0x%"PRIx64"\n",
> p->base);
> fclose(f);
>
> On Sat, Dec 10, 2016 at 3:21 PM, Gopakumar Choorakkot Edakkunni <
> gopakumar@gmail.com> wrote:
>
>> Hi Oliver,
>>
>> Looking at the kernel code pci_mmap_resource()--->pci_mmap_fits(), it
>> looks like pci_mmap_fits() expects that the "number of pages" requested to
>> map is no larger than the actual size of the pci resource. But in
>> pci_uio_ioport_map() what is passed as the requested size is the end_addr
>> of the resource, which in my case is 0xc09f, but the size of the resource
>> itself is 0x20 and hence kernel expects one page - but 0xc09f+1
>> (end_addr+1) gets translated to 13 pages ! And I guess that leads to the
>> mmap to fail .. Thoughts on this ?
>>
>> Rgds,
>> Gopa.
>>
>>
>> On Fri, Dec 9, 2016 at 8:30 PM, Gopakumar Choorakkot Edakkunni <
>> gopakumar@gmail.com> wrote:
>>
>>> I tried to load up dpdk (bound to uio_pci_generic, tried igb_uio also!)
>>> on a virtio interface. Dpdk exits with the "Cannot mmap IO port resource"
>>> complaint - the call chain is
>>> legacy_virtio_resource_init()-->pci_uio_ioport_map().
>>>
>>>
>>> If anyone can give any clues/pointers as to what might be wrong, that
>>> can greatly help in getting me on the right track fast !
>>>
>>> ---
>>> EAL: probe driver: 1af4:1000 rte_virtio_pmd
>>> EAL: PCI device :00:05.0 on NUMA socket 0
>>> EAL: probe driver: 1af4:1000 r[ 330.753458] [ cut here
>>> ]
>>> rte_virtio_pmd
>>> [ 330.754430] WARNING: CPU: 0 PID: 8063 at drivers/pci/pci-sysfs.c:984
>>> pci_mmap_resource.isra.14+0x113/0x1b0()
>>> [ 330.756083] process "dpdk_master" tried to map 0xd000 bytes at
>>> page 0x on :00:05.0 BAR 0 (start 0xc080, size
>>> 0x 20)
>>> [ 330.758272] Modules linked in: rte_kni igb_uio pppoe ppp_async
>>> iptable_nat cdc_mbim sierra_net rndis_host qmi_wwan qcserial pppox
>>> ppp_generic option nf_nat_pptp nf_nat_ipv4 nf_nat_amanda nf_conntrack_pptp
>>> nf_conntrack_ipv4 nf_conntrack_amanda ipt_MASQUERADE huawei_cdc_ncm
>>> cdc_subset cdc_ncm cdc_ether cdc_eem ax88179_178a asix xt_u32 xt_time
>>> xt_tcpmss xt_string xt_statistic xt_state xt_socket xt_recent xt_quota
>>> xt_pkttype xt_physdev xt_owner xt_nat xt_multiport xt_mark xt_mac xt_limit
>>> xt_length xt_iprange xt_id xt_hl xt_helper xt_hashlimit xt_esp xt_ecn
>>> xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment
>>> xt_addrtype xt_TPROXY xt_TEE xt_TCPMSS xt_REDIRECT xt_NETMAP xt_LOG xt_LED
>>> xt_HL xt_DSCP xt_CT xt_CLASSIFY usbnet usb_wwan ts_kmp ts_fsm ts_bm slhc
>>> sierra pl2303 nfnetlink_queue nfnetlink_log nf_nat_tftp nf_nat_snmp_basic
>>> nf_nat_sip nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat
>>> nf_defrag_ipv4 nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip
>>> nf_conntrack_proto_gre nf_conntrack_netlink nf_
Re: [dpdk-dev] dpdk16.07 and virtio "Cannot mmap IO port resource"
Somewhere I read that the "offset" parameter for sysfs mmap() needs to be
the start address of the resource, so I tried the below diff, and mmap
still fails .. Still scratching my head
diff --git a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
index 1786b75..e882940 100644
--- a/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
+++ b/lib/librte_eal/linuxapp/eal/eal_pci_uio.c
@@ -462,8 +462,8 @@ pci_uio_ioport_map(struct rte_pci_device *dev, int bar,
strerror(errno));
goto error;
}
- addr = mmap(NULL, end_addr + 1, PROT_READ | PROT_WRITE,
- MAP_SHARED, fd, 0);
+ addr = mmap(NULL, (end_addr-phys_addr) + 1, PROT_READ | PROT_WRITE,
+ MAP_SHARED, fd, phys_addr);
close(fd);
if (addr == MAP_FAILED) {
RTE_LOG(ERR, EAL, "Cannot mmap IO port resource: %s\n",
@@ -473,7 +473,7 @@ pci_uio_ioport_map(struct rte_pci_device *dev, int bar,
/* strangely, the base address is mmap addr + phys_addr */
p->base = (uintptr_t)addr + phys_addr;
- p->len = end_addr + 1;
+ p->len = (end_addr-phys_addr) + 1;
RTE_LOG(DEBUG, EAL, "PCI Port IO found start=0x%"PRIx64"\n",
p->base);
fclose(f);
On Sat, Dec 10, 2016 at 3:21 PM, Gopakumar Choorakkot Edakkunni <
gopakumar@gmail.com> wrote:
> Hi Oliver,
>
> Looking at the kernel code pci_mmap_resource()--->pci_mmap_fits(), it
> looks like pci_mmap_fits() expects that the "number of pages" requested to
> map is no larger than the actual size of the pci resource. But in
> pci_uio_ioport_map() what is passed as the requested size is the end_addr
> of the resource, which in my case is 0xc09f, but the size of the resource
> itself is 0x20 and hence kernel expects one page - but 0xc09f+1
> (end_addr+1) gets translated to 13 pages ! And I guess that leads to the
> mmap to fail .. Thoughts on this ?
>
> Rgds,
> Gopa.
>
>
> On Fri, Dec 9, 2016 at 8:30 PM, Gopakumar Choorakkot Edakkunni <
> gopakumar@gmail.com> wrote:
>
>> I tried to load up dpdk (bound to uio_pci_generic, tried igb_uio also!)
>> on a virtio interface. Dpdk exits with the "Cannot mmap IO port resource"
>> complaint - the call chain is
>> legacy_virtio_resource_init()-->pci_uio_ioport_map().
>>
>>
>> If anyone can give any clues/pointers as to what might be wrong, that can
>> greatly help in getting me on the right track fast !
>>
>> ---
>> EAL: probe driver: 1af4:1000 rte_virtio_pmd
>> EAL: PCI device :00:05.0 on NUMA socket 0
>> EAL: probe driver: 1af4:1000 r[ 330.753458] [ cut here
>> ]
>> rte_virtio_pmd
>> [ 330.754430] WARNING: CPU: 0 PID: 8063 at drivers/pci/pci-sysfs.c:984
>> pci_mmap_resource.isra.14+0x113/0x1b0()
>> [ 330.756083] process "dpdk_master" tried to map 0xd000 bytes at
>> page 0x on :00:05.0 BAR 0 (start 0xc080, size
>> 0x 20)
>> [ 330.758272] Modules linked in: rte_kni igb_uio pppoe ppp_async
>> iptable_nat cdc_mbim sierra_net rndis_host qmi_wwan qcserial pppox
>> ppp_generic option nf_nat_pptp nf_nat_ipv4 nf_nat_amanda nf_conntrack_pptp
>> nf_conntrack_ipv4 nf_conntrack_amanda ipt_MASQUERADE huawei_cdc_ncm
>> cdc_subset cdc_ncm cdc_ether cdc_eem ax88179_178a asix xt_u32 xt_time
>> xt_tcpmss xt_string xt_statistic xt_state xt_socket xt_recent xt_quota
>> xt_pkttype xt_physdev xt_owner xt_nat xt_multiport xt_mark xt_mac xt_limit
>> xt_length xt_iprange xt_id xt_hl xt_helper xt_hashlimit xt_esp xt_ecn
>> xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment
>> xt_addrtype xt_TPROXY xt_TEE xt_TCPMSS xt_REDIRECT xt_NETMAP xt_LOG xt_LED
>> xt_HL xt_DSCP xt_CT xt_CLASSIFY usbnet usb_wwan ts_kmp ts_fsm ts_bm slhc
>> sierra pl2303 nfnetlink_queue nfnetlink_log nf_nat_tftp nf_nat_snmp_basic
>> nf_nat_sip nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat
>> nf_defrag_ipv4 nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip
>> nf_conntrack_proto_gre nf_conntrack_netlink nf_conntrack_irc
>> nf_conntrack_h323 nf_conntrack_ftp nf_conntrack_broadcast
>> lib80211_crypt_wep lib80211_crypt_tkip lib80211_crypt_ccmp lib80211
>> iptable_raw iptable_mangle iptable_filter ipt_ah ipt_ULOG ipt_REJECT
>> ipt_ECN hso ftdi_sio crc8 crc7 crc_itu_t crc_ccitt cp210x cordic
>> compat_xtables cdc_wdm cdc_acm arptable_filter arpt_mangle arp_tables
>> sch_teql sch_tbf sch_sfq sch_red sch_prio sch_netem sch_htb sch_gred
>> sch_dsmark sch_codel em_text em_nbyte em_meta em_cmp cls_basic act_police
>> act_ipt act_connmark act_skbedit act_mir
Re: [dpdk-dev] dpdk16.07 and virtio "Cannot mmap IO port resource"
Hi Oliver, Looking at the kernel code pci_mmap_resource()--->pci_mmap_fits(), it looks like pci_mmap_fits() expects that the "number of pages" requested to map is no larger than the actual size of the pci resource. But in pci_uio_ioport_map() what is passed as the requested size is the end_addr of the resource, which in my case is 0xc09f, but the size of the resource itself is 0x20 and hence kernel expects one page - but 0xc09f+1 (end_addr+1) gets translated to 13 pages ! And I guess that leads to the mmap to fail .. Thoughts on this ? Rgds, Gopa. On Fri, Dec 9, 2016 at 8:30 PM, Gopakumar Choorakkot Edakkunni < gopakumar@gmail.com> wrote: > I tried to load up dpdk (bound to uio_pci_generic, tried igb_uio also!) on > a virtio interface. Dpdk exits with the "Cannot mmap IO port resource" > complaint - the call chain is > legacy_virtio_resource_init()-->pci_uio_ioport_map(). > > > If anyone can give any clues/pointers as to what might be wrong, that can > greatly help in getting me on the right track fast ! > > --- > EAL: probe driver: 1af4:1000 rte_virtio_pmd > EAL: PCI device :00:05.0 on NUMA socket 0 > EAL: probe driver: 1af4:1000 r[ 330.753458] [ cut here > ] > rte_virtio_pmd > [ 330.754430] WARNING: CPU: 0 PID: 8063 at drivers/pci/pci-sysfs.c:984 > pci_mmap_resource.isra.14+0x113/0x1b0() > [ 330.756083] process "dpdk_master" tried to map 0xd000 bytes at page > 0x on :00:05.0 BAR 0 (start 0xc080, size > 0x 20) > [ 330.758272] Modules linked in: rte_kni igb_uio pppoe ppp_async > iptable_nat cdc_mbim sierra_net rndis_host qmi_wwan qcserial pppox > ppp_generic option nf_nat_pptp nf_nat_ipv4 nf_nat_amanda nf_conntrack_pptp > nf_conntrack_ipv4 nf_conntrack_amanda ipt_MASQUERADE huawei_cdc_ncm > cdc_subset cdc_ncm cdc_ether cdc_eem ax88179_178a asix xt_u32 xt_time > xt_tcpmss xt_string xt_statistic xt_state xt_socket xt_recent xt_quota > xt_pkttype xt_physdev xt_owner xt_nat xt_multiport xt_mark xt_mac xt_limit > xt_length xt_iprange xt_id xt_hl xt_helper xt_hashlimit xt_esp xt_ecn > xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment > xt_addrtype xt_TPROXY xt_TEE xt_TCPMSS xt_REDIRECT xt_NETMAP xt_LOG xt_LED > xt_HL xt_DSCP xt_CT xt_CLASSIFY usbnet usb_wwan ts_kmp ts_fsm ts_bm slhc > sierra pl2303 nfnetlink_queue nfnetlink_log nf_nat_tftp nf_nat_snmp_basic > nf_nat_sip nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat > nf_defrag_ipv4 nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip > nf_conntrack_proto_gre nf_conntrack_netlink nf_conntrack_irc > nf_conntrack_h323 nf_conntrack_ftp nf_conntrack_broadcast > lib80211_crypt_wep lib80211_crypt_tkip lib80211_crypt_ccmp lib80211 > iptable_raw iptable_mangle iptable_filter ipt_ah ipt_ULOG ipt_REJECT > ipt_ECN hso ftdi_sio crc8 crc7 crc_itu_t crc_ccitt cp210x cordic > compat_xtables cdc_wdm cdc_acm arptable_filter arpt_mangle arp_tables > sch_teql sch_tbf sch_sfq sch_red sch_prio sch_netem sch_htb sch_gred > sch_dsmark sch_codel em_text em_nbyte em_meta em_cmp cls_basic act_police > act_ipt act_connmark act_skbedit act_mirred em_u32 cls_u32 cls_tcindex > cls_flow cls_route cls_fw sch_hfsc sch_ingress evdev xt_NFQUEUE xt_set > ip_set_list_set ip_set_hash_netport ip_set_hash_netiface ip_set_hash_net > ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport > ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip > ip_set nfnetlink sr_mod cdrom ip6t_REJECT ip6table_raw ip6table_mangle > ip6table_filter ip6_tables nf_conntrack_ipv6 nf_conntrack nf_defrag_ipv6 > ifb ipip ip6_tunnel tunnel6 tunnel4 ip_tunnel tun mdio loop vfat fat isofs > nls_utf8 nls_iso8859_15 nls_iso8859_1 nls_cp850 nls_cp437 rfkill virtio_rng > sha512_generic sha256_generic sha1_generic md5 arc4 ath10k_pci ath10k_core > ath9k ath9k_common ath9k_hw ath mac80211 cfg80211 compat pcnet32 e1000 > virtio_net ohci_hcd ixgbevf ixgbe hwmon igbvf igb dsa_core mii [last > unloaded: igb_uio] > [ 330.805110] CPU: 0 PID: 8063 Comm: dpdk_master Tainted: GW > 3.14.67 #1 > [ 330.806104] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS > Bochs 01/01/2011 > [ 330.806104] 0286 814e167c > 0006 > [ 330.806104] 88008b573cd8 81259d73 81061b72 > 88013b24fae0 > [ 330.806104] 88013b24f800 88013a498000 > 88013b24f8a8 > [ 330.806104] Call Trace: > [ 330.806104] [] ? dump_stack+0x5e/0x7a > [ 330.806104] [] ? pci_mmap_resource.isra.14+0x11 > 3/0x1b0 > [ 330.806104] [] ? warn_slowpath_common+0x82/0xb0 > [ 330.806104] [] ? warn_slowpath_fmt+0x45/0x50 > [ 330.806104] [] ? do_last.isra.58+0xac1/0xc0
[dpdk-dev] dpdk16.07 and virtio "Cannot mmap IO port resource"
I tried to load up dpdk (bound to uio_pci_generic, tried igb_uio also!) on a virtio interface. Dpdk exits with the "Cannot mmap IO port resource" complaint - the call chain is legacy_virtio_resource_init()-->pci_uio_ioport_map(). If anyone can give any clues/pointers as to what might be wrong, that can greatly help in getting me on the right track fast ! --- EAL: probe driver: 1af4:1000 rte_virtio_pmd EAL: PCI device :00:05.0 on NUMA socket 0 EAL: probe driver: 1af4:1000 r[ 330.753458] [ cut here ] rte_virtio_pmd [ 330.754430] WARNING: CPU: 0 PID: 8063 at drivers/pci/pci-sysfs.c:984 pci_mmap_resource.isra.14+0x113/0x1b0() [ 330.756083] process "dpdk_master" tried to map 0xd000 bytes at page 0x on :00:05.0 BAR 0 (start 0xc080, size 0x 20) [ 330.758272] Modules linked in: rte_kni igb_uio pppoe ppp_async iptable_nat cdc_mbim sierra_net rndis_host qmi_wwan qcserial pppox ppp_generic option nf_nat_pptp nf_nat_ipv4 nf_nat_amanda nf_conntrack_pptp nf_conntrack_ipv4 nf_conntrack_amanda ipt_MASQUERADE huawei_cdc_ncm cdc_subset cdc_ncm cdc_ether cdc_eem ax88179_178a asix xt_u32 xt_time xt_tcpmss xt_string xt_statistic xt_state xt_socket xt_recent xt_quota xt_pkttype xt_physdev xt_owner xt_nat xt_multiport xt_mark xt_mac xt_limit xt_length xt_iprange xt_id xt_hl xt_helper xt_hashlimit xt_esp xt_ecn xt_dscp xt_conntrack xt_connmark xt_connlimit xt_connbytes xt_comment xt_addrtype xt_TPROXY xt_TEE xt_TCPMSS xt_REDIRECT xt_NETMAP xt_LOG xt_LED xt_HL xt_DSCP xt_CT xt_CLASSIFY usbnet usb_wwan ts_kmp ts_fsm ts_bm slhc sierra pl2303 nfnetlink_queue nfnetlink_log nf_nat_tftp nf_nat_snmp_basic nf_nat_sip nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat nf_defrag_ipv4 nf_conntrack_tftp nf_conntrack_snmp nf_conntrack_sip nf_conntrack_proto_gre nf_conntrack_netlink nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp nf_conntrack_broadcast lib80211_crypt_wep lib80211_crypt_tkip lib80211_crypt_ccmp lib80211 iptable_raw iptable_mangle iptable_filter ipt_ah ipt_ULOG ipt_REJECT ipt_ECN hso ftdi_sio crc8 crc7 crc_itu_t crc_ccitt cp210x cordic compat_xtables cdc_wdm cdc_acm arptable_filter arpt_mangle arp_tables sch_teql sch_tbf sch_sfq sch_red sch_prio sch_netem sch_htb sch_gred sch_dsmark sch_codel em_text em_nbyte em_meta em_cmp cls_basic act_police act_ipt act_connmark act_skbedit act_mirred em_u32 cls_u32 cls_tcindex cls_flow cls_route cls_fw sch_hfsc sch_ingress evdev xt_NFQUEUE xt_set ip_set_list_set ip_set_hash_netport ip_set_hash_netiface ip_set_hash_net ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink sr_mod cdrom ip6t_REJECT ip6table_raw ip6table_mangle ip6table_filter ip6_tables nf_conntrack_ipv6 nf_conntrack nf_defrag_ipv6 ifb ipip ip6_tunnel tunnel6 tunnel4 ip_tunnel tun mdio loop vfat fat isofs nls_utf8 nls_iso8859_15 nls_iso8859_1 nls_cp850 nls_cp437 rfkill virtio_rng sha512_generic sha256_generic sha1_generic md5 arc4 ath10k_pci ath10k_core ath9k ath9k_common ath9k_hw ath mac80211 cfg80211 compat pcnet32 e1000 virtio_net ohci_hcd ixgbevf ixgbe hwmon igbvf igb dsa_core mii [last unloaded: igb_uio] [ 330.805110] CPU: 0 PID: 8063 Comm: dpdk_master Tainted: GW 3.14.67 #1 [ 330.806104] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011 [ 330.806104] 0286 814e167c 0006 [ 330.806104] 88008b573cd8 81259d73 81061b72 88013b24fae0 [ 330.806104] 88013b24f800 88013a498000 88013b24f8a8 [ 330.806104] Call Trace: [ 330.806104] [] ? dump_stack+0x5e/0x7a [ 330.806104] [] ? pci_mmap_resource.isra.14+0x113/0x1b0 [ 330.806104] [] ? warn_slowpath_common+0x82/0xb0 [ 330.806104] [] ? warn_slowpath_fmt+0x45/0x50 [ 330.806104] [] ? do_last.isra.58+0xac1/0xc00 [ 330.806104] [] ? pci_mmap_resource.isra.14+0x113/0x1b0 [ 330.806104] [] ? kernfs_fop_mmap+0x69/0x100 [ 330.806104] [] ? mmap_region+0x2e8/0x560 [ 330.806104] [] ? do_mmap_pgoff+0x318/0x3b0 [ 330.806104] [] ? vm_mmap_pgoff+0x74/0xc0 [ 330.806104] [] ? SyS_mmap_pgoff+0x18a/0x1e0 [ 330.806104] [] ? system_call_fastpath+0x16/0x1b [ 330.823897] ---[ end trace c6106b67c768d501 ]--- EAL: Cannot mmap IO port resource: Invalid argument EAL: Error - exiting with code: 1 Cause: Requested device :00:05.0 cannot be used
[dpdk-dev] dpdk 16.07, issues with rte_mempool_create and rte_kni_alloc()
Thank you Ferruh, I will give this a spin over the weekend and let you know. Rgds, Gopa. On Thu, Aug 25, 2016 at 6:51 AM, Ferruh Yigit wrote: > On 8/10/2016 11:51 AM, Ferruh Yigit wrote: > > Hi Gopakumar, > > > > On 8/4/2016 5:14 PM, Ferruh Yigit wrote: > >> On 8/1/2016 10:19 PM, Gopakumar Choorakkot Edakkunni wrote: > >>> Well, for my purpose I just ended up creating a seperate/smaller pool > >>> earlier during bootup to try to guarantee its from one memseg. > >>> > >>> But I am assuming that this KNI restriction is something thats > "currently" > >>> not fixed and is "fixable" ? > >> > >> > >>> Any ideas on what the summary of the reason > >>> for this restriction is - I was gonna check if I can fix that > >> > >> KNI expects all mbufs are from a physically continuous memory. This is > >> because of current address translation implementation. > >> > >> mbufs allocated in userspace and accessed from both user and kernel > >> space, so mbuf userspace virtual address needs to be converted into > >> kernelspace virtual address. > >> > >> Currently this address translation done by first calculating an offset > >> between virtual addresses using first filed of mempool, later applying > >> same offset to all mbufs. This is why all mbufs should be in physically > >> continuous memory. > >> > >> I think this address translation can be done in different way which can > >> remove the restriction, but not sure about the effect of the > >> performance. I will send a patch for this. > > > > I have sent a patch to remove KNI restriction: > > http://dpdk.org/dev/patchwork/patch/15171/ > > > > Can you please test this patch with a mempool with multiple memzone? > > You need to remove following check in KNI manually: > > if (mp->nb_mem_chunks != 1) > > goto kni_fail; > > Hi Gopakumar, > > Off the list. > > Any chance to test this? > > Thanks, > ferruh > >
[dpdk-dev] SR-IOV: dpdk16.07 on guest and ixgbe 3.19.1 on host - will that be a problem ? (requested invalid api version)
I see the below message "requested invalid api version" on host ? Will that be an issue or is it benign ? Should I upgrade the host ixgbe to a later version ? On the host: *[69576.199458] ixgbe :88:00.1 p4p2: VF 2 requested invalid api version 3* root# lsb_release -a No LSB modules are available. Distributor ID:Ubuntu Description:Ubuntu 14.04.4 LTS Release:14.04 Codename:trusty root at mtpnjrsv113:~# root# ethtool -i p4p2 driver: ixgbe version: 3.19.1 firmware-version: 0x8811, 1.1067.0 bus-info: :88:00.1 supports-statistics: yes supports-test: yes supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: no On the guest: root# lsb_release -a No LSB modules are available. Distributor ID:Ubuntu Description:Ubuntu 14.04.2 LTS Release:14.04 Codename:trusty root# find / -name rte_kni.ko /var/lib/dkms/dpdk/16.07/3.16.0-30-generic/x86_64/module/rte_kni.ko /lib/modules/3.16.0-30-generic/updates/dkms/rte_kni.ko
[dpdk-dev] dpdk 16.07, issues with rte_mempool_create and rte_kni_alloc()
Well, for my purpose I just ended up creating a seperate/smaller pool earlier during bootup to try to guarantee its from one memseg. But I am assuming that this KNI restriction is something thats "currently" not fixed and is "fixable" ? Any ideas on what the summary of the reason for this restriction is - I was gonna check if I can fix that Rgds, Gopa. On Sat, Jul 30, 2016 at 10:39 PM, Gopakumar Choorakkot Edakkunni < gopakumar.c.e at gmail.com> wrote: > Hi all, > > So 16.07 seems to have an rte_mempool_create() which can allocate items > from multiple memzones (commit d1d914eb), which is awesome and is the > reason I upgraed to 16.07 !! > > But .. when I call rte_kni_alloc() with the mempool as parameter, theres a > check in there which says below > > /* KNI currently requires to have only one memory chunk */ > if (mp->nb_mem_chunks != 1) > goto kni_fail; > > And I hit that check because I am allocating a large mempool that happens > to span multiple memzones. I am perfectly fine with allocating a seperate > mempool just for KNI, except that I am unable to find any API or a flag or > some setting which says "create me a mempool that doesnt span memzones", > just so that I can use it with rte_kni_alloc() > > Any suggestions will be helpful ! > > Rgds, > Gopa. > >
[dpdk-dev] dpdk 16.07, issues with rte_mempool_create and rte_kni_alloc()
Hi all, So 16.07 seems to have an rte_mempool_create() which can allocate items from multiple memzones (commit d1d914eb), which is awesome and is the reason I upgraed to 16.07 !! But .. when I call rte_kni_alloc() with the mempool as parameter, theres a check in there which says below /* KNI currently requires to have only one memory chunk */ if (mp->nb_mem_chunks != 1) goto kni_fail; And I hit that check because I am allocating a large mempool that happens to span multiple memzones. I am perfectly fine with allocating a seperate mempool just for KNI, except that I am unable to find any API or a flag or some setting which says "create me a mempool that doesnt span memzones", just so that I can use it with rte_kni_alloc() Any suggestions will be helpful ! Rgds, Gopa.
[dpdk-dev] ethtool doesnt work on some interface after unbinding dpdk
Just to update this thread. With power management for IGB disabled, this problem is not seen .. So for now thats the "workaround" Rgds, Gopa. On Mon, Apr 18, 2016 at 3:08 AM, Remy Horton wrote: > > On 15/04/2016 23:56, Gopakumar Choorakkot Edakkunni wrote: > >> This time the problem statement is more narrowed down. >> >> 1. dpdk is enabled on the interface, interfaces bound to igb_uio >> 3. kill the process using dpdk >> 3. rmmod rte_kni >> 4. rmmod igb_uio >> 5. bind interface to igb >> 6. ethtool, ifconfig up/down etc.. works for approximately 30 seconds, >> and then stops working >> > > Hmm.. can you try that but with rte_kni left out completely? KNI hooks > into the Linux network stack and think it at the least needs eliminating as > a casual factor. Can you also try using uio_pci_generic rather than igb_uio? > > Those aside, I'm suspecting driver issues, so seeing if I can get one of > the driver test guys to have a look at this.. > > > Regards, > > ..Remy >
[dpdk-dev] ethtool doesnt work on some interface after unbinding dpdk
Well, I jumped to a conclusion too soon on the ACPI, that was a wrong statement (wishful thinking), I recreated the issue even with ACPI off This time the problem statement is more narrowed down. 1. dpdk is enabled on the interface, interfaces bound to igb_uio 3. kill the process using dpdk 3. rmmod rte_kni 4. rmmod igb_uio 5. bind interface to igb 6. ethtool, ifconfig up/down etc.. works for approximately 30 seconds, and then stops working At step #6, if I do an lspci of the device, the device is completely shut down (attached a sample lspci output - memory not initialized, irqs not present in /proc/interrupts etc..). But theres nothing in the dmesg that shows any kind of errors after the messages about the interface being bound in step #5 And the wierd part is that the device is up at step #6 for like 30 seconds before it appears shut down in the pci output. Another observation is that in step #4 when igb_uio relinquishes control of the device it still seems to leave it in an initialized state, so the only theory I can think of is whether the device being left in an initia;ized state when it was handed off to igb in step #5 caused igb to run into some error and shut down the device ? The next step is to enable debugs in igb driver and see if there are any debugs that tells more about what happened Rgds, Gopa. On Fri, Apr 15, 2016 at 12:31 PM, Gopakumar Choorakkot Edakkunni < gopakumar.c.e at gmail.com> wrote: > So looks like I figured it out .. I came across this bug reference > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=728692 and thought of > checking out my problem with ACPI turned off. And with ACPI turned off, the > problem doesnt happen. So theres something that the igb driver is not happy > about when acpi is on .. Any thoughts ? > > Rgds, > Gopa. > > On Fri, Apr 15, 2016 at 10:38 AM, Gopakumar Choorakkot Edakkunni < > gopakumar.c.e at gmail.com> wrote: > >> Nothing in dmesg .. The ethtool was just a side-observation, the biggest >> problem was that after unbinding from igb_uio and rebinding to igb, if I >> follow it up with an /etc/init.d/network restart, that completely hoses the >> linux system - anyone trying to open a socket (ifconfig for example) just >> hangs. Thats how I started troubleshooting this and happened to see this >> ethtool thing along with it, not sure if its related. Also the issue >> doesn't happen with one or two interfaces, there needs to be at least five >> or six interfaces for this to happen. >> >> The other thing I noticed is that if I put some sleep (2 seconds) between >> unbind igb_uio and re-bind igb, the network-restart-hosing-system doesnt >> happen, but the ethtool issue still remains >> >> Rgds, >> Gopa. >> >> On Fri, Apr 15, 2016 at 12:50 AM, Remy Horton >> wrote: >> >>> On 14/04/2016 20:25, Gopakumar Choorakkot Edakkunni wrote: >>> [..] >>> >>>> ge8-> 06:00.0 Ethernet controller: Intel Corporation 82576 Gigabit >>>> Network Connection (rev 01) >>>> >>>> root:~# ls /sys/class/net/ge8/device/driver/module/drivers/ >>>> pci:igb >>>> root:~# >>>> >>>> root:~# ethtool ge8 >>>> Settings for ge8: >>>> Cannot get device settings: No such device >>>> Cannot get wake-on-lan settings: No such device >>>> Cannot get message level: No such device >>>> Cannot get link status: No such device >>>> No data available >>>> >>> >>> Seems a little odd. Does dmesg show anything related to igb/ixgbe when >>> you try this? >>> >>> ..Remy >>> >> >> >
[dpdk-dev] ethtool doesnt work on some interface after unbinding dpdk
So looks like I figured it out .. I came across this bug reference https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=728692 and thought of checking out my problem with ACPI turned off. And with ACPI turned off, the problem doesnt happen. So theres something that the igb driver is not happy about when acpi is on .. Any thoughts ? Rgds, Gopa. On Fri, Apr 15, 2016 at 10:38 AM, Gopakumar Choorakkot Edakkunni < gopakumar.c.e at gmail.com> wrote: > Nothing in dmesg .. The ethtool was just a side-observation, the biggest > problem was that after unbinding from igb_uio and rebinding to igb, if I > follow it up with an /etc/init.d/network restart, that completely hoses the > linux system - anyone trying to open a socket (ifconfig for example) just > hangs. Thats how I started troubleshooting this and happened to see this > ethtool thing along with it, not sure if its related. Also the issue > doesn't happen with one or two interfaces, there needs to be at least five > or six interfaces for this to happen. > > The other thing I noticed is that if I put some sleep (2 seconds) between > unbind igb_uio and re-bind igb, the network-restart-hosing-system doesnt > happen, but the ethtool issue still remains > > Rgds, > Gopa. > > On Fri, Apr 15, 2016 at 12:50 AM, Remy Horton > wrote: > >> On 14/04/2016 20:25, Gopakumar Choorakkot Edakkunni wrote: >> [..] >> >>> ge8-> 06:00.0 Ethernet controller: Intel Corporation 82576 Gigabit >>> Network Connection (rev 01) >>> >>> root:~# ls /sys/class/net/ge8/device/driver/module/drivers/ >>> pci:igb >>> root:~# >>> >>> root:~# ethtool ge8 >>> Settings for ge8: >>> Cannot get device settings: No such device >>> Cannot get wake-on-lan settings: No such device >>> Cannot get message level: No such device >>> Cannot get link status: No such device >>> No data available >>> >> >> Seems a little odd. Does dmesg show anything related to igb/ixgbe when >> you try this? >> >> ..Remy >> > >
[dpdk-dev] ethtool doesnt work on some interface after unbinding dpdk
Nothing in dmesg .. The ethtool was just a side-observation, the biggest problem was that after unbinding from igb_uio and rebinding to igb, if I follow it up with an /etc/init.d/network restart, that completely hoses the linux system - anyone trying to open a socket (ifconfig for example) just hangs. Thats how I started troubleshooting this and happened to see this ethtool thing along with it, not sure if its related. Also the issue doesn't happen with one or two interfaces, there needs to be at least five or six interfaces for this to happen. The other thing I noticed is that if I put some sleep (2 seconds) between unbind igb_uio and re-bind igb, the network-restart-hosing-system doesnt happen, but the ethtool issue still remains Rgds, Gopa. On Fri, Apr 15, 2016 at 12:50 AM, Remy Horton wrote: > On 14/04/2016 20:25, Gopakumar Choorakkot Edakkunni wrote: > [..] > >> ge8-> 06:00.0 Ethernet controller: Intel Corporation 82576 Gigabit >> Network Connection (rev 01) >> >> root:~# ls /sys/class/net/ge8/device/driver/module/drivers/ >> pci:igb >> root:~# >> >> root:~# ethtool ge8 >> Settings for ge8: >> Cannot get device settings: No such device >> Cannot get wake-on-lan settings: No such device >> Cannot get message level: No such device >> Cannot get link status: No such device >> No data available >> > > Seems a little odd. Does dmesg show anything related to igb/ixgbe when you > try this? > > ..Remy >
[dpdk-dev] ethtool doesnt work on some interface after unbinding dpdk
Hi Remy, Thanks for the response. The error is "No such device", some snippets below. And no I was not using the dpdk ethtool ge8-> 06:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01) root:~# ls /sys/class/net/ge8/device/driver/module/drivers/ pci:igb root:~# root:~# ethtool ge8 Settings for ge8: Cannot get device settings: No such device Cannot get wake-on-lan settings: No such device Cannot get message level: No such device Cannot get link status: No such device No data available Rgds, Gopa. On Thu, Apr 14, 2016 at 12:15 AM, Remy Horton wrote: > Morning, > > On 13/04/2016 15:48, Gopakumar Choorakkot Edakkunni wrote: > [..] > >> then after a while I >> unbind from igb_uio and bind them back to igb/ixgbe. At this point, one of >> the 4 igb ports (random) stops responding to ethtool, ethtool bails out >> with some error. But otherwise the interface seems to work fine, it has a >> linux interface created and pops up in /sys/class/net etc.. Has anyone >> seen >> this before ? I thought of checking before starting to debug this further >> > > Can you give details of the error? If you were you unbinding from igb_uio > while examples/ethtool was still running it likley caused something to trip > up, as at least DPDK ethtool itself was not made with run-time unbinding in > mind. > > Regards, > > ..Remy >
[dpdk-dev] ethtool doesnt work on some interface after unbinding dpdk
All, I have a Xeon machine with linux kernel 3.14 and dpdk 2.0, I have 6 ports of which 4 are igb and 2 are ixgbe drivers .. I use the python script in the dpdk distribution to bind and unbind the ports. I unbind them from linux and bind to igb_uio, dpdk works just fine, then after a while I unbind from igb_uio and bind them back to igb/ixgbe. At this point, one of the 4 igb ports (random) stops responding to ethtool, ethtool bails out with some error. But otherwise the interface seems to work fine, it has a linux interface created and pops up in /sys/class/net etc.. Has anyone seen this before ? I thought of checking before starting to debug this further Rgds, Gopa.
[dpdk-dev] Performance of rte_ring APIs
Hi All, I have an extremely simple test - I have just one single DPDK EAL thread which pulls packets from one 10G port and just puts the packet exactly as is (no changes) on another 10G port - I get 9.5million pps, so far so good. So its like this dpdk_rx dpdk_tx 9.5 Millionpps Now I do the below and the performance comes down to like 4 Million pps, less than half ! dpdk_rx rte_ring_mc_dequeue_bulk(my_ring1, my_array, nb_rx) dpdk_tx rte_ring_mp_enqueue_bulk(my_ring1, my_array, nb_rx) Note that I do nothing with the things I dequeue from the ring, I just enqueue it back. So is there any gotchas in using these rings ? I am sure I am missing something, it cant drop from 9.5Mpps to 4Mpps just because of a dequeue/enqueue ? Rgds, Gopa.
[dpdk-dev] Diff for persistent KNI interfaces
I guess theres not much interest from others for this support in
general, but putting it out here in case anyone wants to take a look.
If anyone finds any major issues with the diffset, please do let me
know, that will be of great help. I have tested this on my machine.
Rgds,
Gopa.
-- next part --
commit 58968f690056d5afa7517e724bc91f6436639d10
Author: Gopakumar C E
Date: Sun Jul 12 11:18:33 2015 -0700
KNI persistent interfaces
diff --git a/lib/librte_eal/linuxapp/eal/include/exec-env/rte_kni_common.h
b/lib/librte_eal/linuxapp/eal/include/exec-env/rte_kni_common.h
index 1e55c2d..369ab85 100644
--- a/lib/librte_eal/linuxapp/eal/include/exec-env/rte_kni_common.h
+++ b/lib/librte_eal/linuxapp/eal/include/exec-env/rte_kni_common.h
@@ -160,6 +160,7 @@ struct rte_kni_device_info {
uint32_t core_id; /**< core ID to bind for kernel thread */
uint8_t force_bind : 1; /**< Flag for kernel thread binding */
+ uint8_t persist_on_close : 1; /**< Dont delete interface on dev/kni
close */
/* mbuf size */
unsigned mbuf_size;
diff --git a/lib/librte_eal/linuxapp/kni/kni_dev.h
b/lib/librte_eal/linuxapp/kni/kni_dev.h
index e79e472..e50aad7 100644
--- a/lib/librte_eal/linuxapp/kni/kni_dev.h
+++ b/lib/librte_eal/linuxapp/kni/kni_dev.h
@@ -96,6 +96,12 @@ struct kni_dev {
/* synchro for request processing */
unsigned long synchro;
+ /* Persist netdev on /dev/kni close */
+ uint8_t netdev_persist : 1;
+
+ /* /dev/kni is closed or not */
+ uint8_t kni_released : 1;
+
#ifdef RTE_KNI_VHOST
struct kni_vhost_queue* vhost_queue;
volatile enum {
diff --git a/lib/librte_eal/linuxapp/kni/kni_misc.c
b/lib/librte_eal/linuxapp/kni/kni_misc.c
index 1935d32..f386a31 100644
--- a/lib/librte_eal/linuxapp/kni/kni_misc.c
+++ b/lib/librte_eal/linuxapp/kni/kni_misc.c
@@ -58,7 +58,7 @@ static int kni_ioctl(struct inode *inode, unsigned int
ioctl_num,
unsigned long ioctl_param);
static int kni_compat_ioctl(struct inode *inode, unsigned int ioctl_num,
unsigned long ioctl_param);
-static int kni_dev_remove(struct kni_dev *dev);
+static int kni_dev_remove(struct kni_dev *dev, int netdev_remove);
static int __init kni_parse_kthread_mode(void);
@@ -128,6 +128,21 @@ kni_init(void)
static void __exit
kni_exit(void)
{
+struct kni_dev *dev, *n;
+
+ /*
+* Remove all the persistent KNI interfaces.
+*/
+down_write(&kni_list_lock);
+list_for_each_entry_safe(dev, n, &kni_list_head, list) {
+ if (dev->net_dev) {
+ unregister_netdev(dev->net_dev);
+ free_netdev(dev->net_dev);
+ }
+list_del(&dev->list);
+ }
+up_write(&kni_list_lock);
+
misc_deregister(&kni_misc);
KNI_PRINT("### DPDK kni module unloaded ###\n");
}
@@ -196,8 +211,12 @@ kni_release(struct inode *inode, struct file *file)
#ifdef RTE_KNI_VHOST
kni_vhost_backend_release(dev);
#endif
- kni_dev_remove(dev);
- list_del(&dev->list);
+ kni_dev_remove(dev, !dev->netdev_persist);
+ if (!dev->netdev_persist) {
+ list_del(&dev->list);
+ } else {
+ dev->kni_released = 1;
+ }
}
up_write(&kni_list_lock);
@@ -264,7 +283,7 @@ kni_thread_multiple(void *param)
}
static int
-kni_dev_remove(struct kni_dev *dev)
+kni_dev_remove(struct kni_dev *dev, int netdev_remove)
{
if (!dev)
return -ENODEV;
@@ -282,7 +301,7 @@ kni_dev_remove(struct kni_dev *dev)
break;
}
- if (dev->net_dev) {
+ if (dev->net_dev && netdev_remove) {
unregister_netdev(dev->net_dev);
free_netdev(dev->net_dev);
}
@@ -314,8 +333,9 @@ kni_ioctl_create(unsigned int ioctl_num, unsigned long
ioctl_param)
struct pci_dev *found_pci = NULL;
struct net_device *net_dev = NULL;
struct net_device *lad_dev = NULL;
- struct kni_dev *kni, *dev, *n;
+ struct kni_dev *kni = NULL, *dev, *n;
struct net *net;
+ int kni_exists = 0;
printk(KERN_INFO "KNI: Creating kni...\n");
/* Check the buffer size, to avoid warning */
@@ -343,32 +363,52 @@ kni_ioctl_create(unsigned int ioctl_num, unsigned long
ioctl_param)
down_read(&kni_list_lock);
list_for_each_entry_safe(dev, n, &kni_list_head, list) {
if (kni_check_param(dev, &dev_info) < 0) {
- up_read(&kni_list_lock);
- return -EINVAL;
+ kni = dev;
+ kni_exists = 1;
+ break;
}
}
[dpdk-dev] How to prevent KNI interface from getting deleted on application termination?
You are right, I dint think of the rings. I guess if I keep the netdev intact and release the rings and stuff as it does on a /dev/kni close, that should be fine ? And when my app comes back up and opens dev/kni, the rings etc.. can be recreated, and when my app tries to create the netdevs, it reattaches it to the existing ones. The way I look at it, its similar to the tun/tap ioctl option TUNSETPERSIST. Rgds, Gopa. On Thu, Jul 9, 2015 at 9:46 AM, Stephen Hemminger wrote: > On Wed, 8 Jul 2015 23:36:52 -0700 > Gopakumar Choorakkot Edakkunni wrote: > >> Reading through the KNI module source, doesnt look like there is a way >> to do this. For my requirement, I will make some patch tomorrow to >> have a module option to just keep the KNI data structures around even >> if /dev/kni is closed, looks straightforward to do from the code > > > I don't think it is that simple given the shared ring in KNI
[dpdk-dev] How to prevent KNI interface from getting deleted on application termination?
Crash is just one example I called out, the other example is a hotfix update to the app which ends up gracefully restarting the app. Rgds, Gopa. On Thu, Jul 9, 2015 at 12:47 AM, Marc Sune wrote: > > > On 09/07/15 08:36, Gopakumar Choorakkot Edakkunni wrote: >> >> Reading through the KNI module source, doesnt look like there is a way >> to do this. For my requirement, I will make some patch tomorrow to >> have a module option to just keep the KNI data structures around even >> if /dev/kni is closed, looks straightforward to do from the code >> >> Rgds, >> Gopa. >> >> On Wed, Jul 8, 2015 at 1:00 PM, Gopakumar Choorakkot Edakkunni >> wrote: >>> >>> Hi all, >>> >>> My application takes over one/multiple ethernet port(s) in a linux >>> system and creates KNI interfaces corresponding to them. So if there >>> was eth0 and eth1 in the non-dpdk mode, once I take over the ports >>> using dpdk, I create eth0 and eth1 KNI interfaces. As far as the linux >>> network managers are concerned, they dont really know about it (or >>> care I guess) - for example the dhcp client tries getting a dhcp >>> address over these KNI interfaces and succeeds. >>> >>> Now if my application crashes, I dont want the entire network >>> management subsystem on linux and the hotplugs and this and that to >>> get alarmed and routes to vanish from the route table etc.. etc.. The >>> application will crash and come back up real quick, nothing needs to >>> change in that meantime. > > > Maybe a stupid question; why not fixing your application so that it doesn't > crash, instead of adding adhoc patches? > > marc > > >>> >>> Any way to achieve that ? I just want to keep the KNI around even if >>> my app vanishes. >>> >>> Rgds, >>> Gopa. > >
[dpdk-dev] How to prevent KNI interface from getting deleted on application termination?
Reading through the KNI module source, doesnt look like there is a way to do this. For my requirement, I will make some patch tomorrow to have a module option to just keep the KNI data structures around even if /dev/kni is closed, looks straightforward to do from the code Rgds, Gopa. On Wed, Jul 8, 2015 at 1:00 PM, Gopakumar Choorakkot Edakkunni wrote: > Hi all, > > My application takes over one/multiple ethernet port(s) in a linux > system and creates KNI interfaces corresponding to them. So if there > was eth0 and eth1 in the non-dpdk mode, once I take over the ports > using dpdk, I create eth0 and eth1 KNI interfaces. As far as the linux > network managers are concerned, they dont really know about it (or > care I guess) - for example the dhcp client tries getting a dhcp > address over these KNI interfaces and succeeds. > > Now if my application crashes, I dont want the entire network > management subsystem on linux and the hotplugs and this and that to > get alarmed and routes to vanish from the route table etc.. etc.. The > application will crash and come back up real quick, nothing needs to > change in that meantime. > > Any way to achieve that ? I just want to keep the KNI around even if > my app vanishes. > > Rgds, > Gopa.
[dpdk-dev] How to prevent KNI interface from getting deleted on application termination?
Hi all, My application takes over one/multiple ethernet port(s) in a linux system and creates KNI interfaces corresponding to them. So if there was eth0 and eth1 in the non-dpdk mode, once I take over the ports using dpdk, I create eth0 and eth1 KNI interfaces. As far as the linux network managers are concerned, they dont really know about it (or care I guess) - for example the dhcp client tries getting a dhcp address over these KNI interfaces and succeeds. Now if my application crashes, I dont want the entire network management subsystem on linux and the hotplugs and this and that to get alarmed and routes to vanish from the route table etc.. etc.. The application will crash and come back up real quick, nothing needs to change in that meantime. Any way to achieve that ? I just want to keep the KNI around even if my app vanishes. Rgds, Gopa.
[dpdk-dev] Using rte_ring_mp_xyz() across EAL and non-EAL threads ?
Thanks for the clarification Bruce. But I find this link below in the documentation which says it should not be used in cases where the scheduling policy is SCHED_RR because guess it can lead to an endless spin-lock-wait by the SCHED_RR thread waiting for the lower prio thread which got pre-empted. I am assuming this is a very unlikely situation 32 core Xeon like cpu where a low prio thread never gets scheduled because there are always realtime threads with work to do ? But since the warning is in bold and pretty loud :), I decided not to use that and just do a handoff with a standard mutex-lock :(. If my threads were not realtime (not SCHED_RR), I could have used this! http://dpdk.org/doc/guides/prog_guide/env_abstraction_layer.html section 3.3.4. "The ?non-preemptive? constraint means: Bypassing this constraint it may cause the 2nd pthread to spin until the 1st one is scheduled again. Moreover, if the 1st pthread is preempted by a context that has an higher priority, it may even cause a dead lock." Rgds, Gopa. On Thu, Jul 2, 2015 at 2:20 AM, Bruce Richardson wrote: > On Wed, Jul 01, 2015 at 10:50:49AM -0700, Gopakumar Choorakkot Edakkunni > wrote: >> rte_ring_create() needs a socket-id though and seems to be allocating >> core-specific memory pools for the ring ? But my non-EAL app thread is >> not bound to any core, so now I am wondering if that will work. >> >> Rgds, >> Gopa. > > There are no core-specific elements for rte_rings, just for mempools. Yes, you > need a NUMA node ID when creating the ring, so that DPDK knows where to > allocate > the memory for it. However, once that is done, the ring can safely be used > from > both EAL and non-EAL threads. There is no requirement to have an lcore-id for > the thread. > > /Bruce > >> >> On Wed, Jul 1, 2015 at 10:46 AM, Gopakumar Choorakkot Edakkunni >> wrote: >> > Hi, >> > >> > I have a requirement where one of my non-EAL app threads needs to >> > handoff some packets to an EAL task. I was thinking of using >> > rte_ring_mp_enqueue/dequeue for that purpose. I looked at the code for >> > the rte_ring library and it doesnt look like it has any "EAL" >> > dependencies, but I wanted to double confirm that there are no issues >> > in using it that way. Dint find much yes/no info about that on the >> > mailers/docs. Pls let me know your thoughts. >> > >> > Rgds, >> > Gopa.
[dpdk-dev] Using rte_ring_mp_xyz() across EAL and non-EAL threads ?
rte_ring_create() needs a socket-id though and seems to be allocating core-specific memory pools for the ring ? But my non-EAL app thread is not bound to any core, so now I am wondering if that will work. Rgds, Gopa. On Wed, Jul 1, 2015 at 10:46 AM, Gopakumar Choorakkot Edakkunni wrote: > Hi, > > I have a requirement where one of my non-EAL app threads needs to > handoff some packets to an EAL task. I was thinking of using > rte_ring_mp_enqueue/dequeue for that purpose. I looked at the code for > the rte_ring library and it doesnt look like it has any "EAL" > dependencies, but I wanted to double confirm that there are no issues > in using it that way. Dint find much yes/no info about that on the > mailers/docs. Pls let me know your thoughts. > > Rgds, > Gopa.
[dpdk-dev] Using rte_ring_mp_xyz() across EAL and non-EAL threads ?
Hi, I have a requirement where one of my non-EAL app threads needs to handoff some packets to an EAL task. I was thinking of using rte_ring_mp_enqueue/dequeue for that purpose. I looked at the code for the rte_ring library and it doesnt look like it has any "EAL" dependencies, but I wanted to double confirm that there are no issues in using it that way. Dint find much yes/no info about that on the mailers/docs. Pls let me know your thoughts. Rgds, Gopa.
[dpdk-dev] dpdk-2.0.0: crash in ixgbe_recv_scattered_pkts_vec->_recv_raw_pkts_vec->desc_to_olflags_v
So update on this. Summary is that its purely my fault, apologies for prematurely suspecting the wrong areas. Details below 1. So my AWS box had an eth0 interface without DPDK, I enabled dpdk AND created a KNI interface also AND named the KNI interface to be eth0 2. So Ubuntu started its dhcpclient on that interface, but my app doesnt really do anything do read the dhcp (renews) from the KNI and send it out the physical port and vice versa .. The kni was just sitting there not doing much of Rx/Tx 3. Now my l2fwd-equivalent code started working fine, after a few minutes, the dhcp client on ubuntu gave up attempting dhcp renew (eth0 already had an IP) and attempted to take off the IP from eth0 4. At this point the standard KNI examples in dpdk which has callbacks registered, ended up being invoked - and the examples have a port_stop() and a port_start() in them - and exactly at this point my app crashed So my bad! I just no-oped the callbacks for now and changed AWS eht0 from dhcp to static IP and this are fine now ! My system has been up for long with no issues. Thanks again Thomas and Bruce for the quick response and suggestions Rgds, Gopa. On Tue, Jun 30, 2015 at 11:28 AM, Gopakumar Choorakkot Edakkunni wrote: > Hi Thomas, Bruce, > > Thanks for the responses. Please find my answers as below. > > Thomas>> "You mean you are using SR-IOV from Amazon, right? Do you > have more hardware details?" > > That is correct. I am attaching three files cpuinfo.txt lcpci.txt and > portconf.txt (just the port config that I am using, nothing special, > yanked off of l2fwd example). The two 82599 VF interfaces seen in > lspci output are the ones of interest - I use one of them in dpdk > mode. > > Thomas>> Did you try to disable CONFIG_RTE_IXGBE_INC_VECTOR? > > Thanks for the suggestion, I made that change and was giving it some > time. Now the result of that is not entirely black and white: > previously (in vector mode) my app used to Rx/Tx packets nicely > without any hiccups, but would crash in 10 minutes :). Now with this > suggested change, its been running for a while and doesnt crash, but > the Tx latency and Tx loss is so high (around 10% tx loss) that the > app is not doing a great job - but that might just be something that I > need to adapt to when using non-vector mode ? I will experiment on > that a bit more. So I "think" its fair to say that with the vector > disabled, theres no crash, but I need to chase this latency/loss now. > > Thomas>> Not needed. A DPDK application is fast enough to do the job > in 10 minutes ;) > > Haha, good one :). Thats where I want to get to eventually, but right > now some distance from it. > > Bruce>> Can you perhaps isolate any further the root cause of the > issue. For example, does it only occur when you get three packets at > the receive ring wraps back around to zero? > > I will try some more experiments, will read and understand this Rx > code a bit more to be able to answer the qn about whether ring wraps > around when the problem happens etc.. > > Rgds, > Gopa. > > > On Tue, Jun 30, 2015 at 9:08 AM, Thomas Monjalon > wrote: >> 2015-06-30 08:49, Gopakumar Choorakkot Edakkunni: >>> I am starting to tryout dpdk-2.0.0 with a simple Rx routine very >>> similar to the l2fwd example - I am running this on a c3.8xlarge aws >>> sr-iov enabled vpc instance (inside the vm it uses ixgbevf driver). >> >> You mean you are using SR-IOV from Amazon, right? >> Do you have more hardware details? >> >>> Once in every 10 minutes my application crashes in the recieve path. >>> And whenever I check the crash reason its because it always has three >>> packets in the burst array (I have provided array size of 32) instead >>> of the four that it tries to collect in one bunch. And inside >>> desc_to_olflags_v(), theres the assumption that there are four >>> packets, and obviously it crashes trying to access the fourth buffer. >> >> Did you try to disable CONFIG_RTE_IXGBE_INC_VECTOR? >> >>> With a brief look at the code, I really cant make out how its >>> guaranteed that we will always have four descriptors fully populated ? >>> After the first iteration, the loop does break out if (likely(var != >>> RTE_IXGBE_DESCS_PER_LOOP)), but how about the very first iteration >>> where we might not have four ? >>> >>> Any thoughts will be helpful here, trying to get my app working for >>> more than 10 minutes :) >> >> Not needed. A DPDK application is fast enough to do the job in 10 minutes ;) >>
[dpdk-dev] dpdk-2.0.0: crash in ixgbe_recv_scattered_pkts_vec->_recv_raw_pkts_vec->desc_to_olflags_v
Hi Thomas, Bruce, Thanks for the responses. Please find my answers as below. Thomas>> "You mean you are using SR-IOV from Amazon, right? Do you have more hardware details?" That is correct. I am attaching three files cpuinfo.txt lcpci.txt and portconf.txt (just the port config that I am using, nothing special, yanked off of l2fwd example). The two 82599 VF interfaces seen in lspci output are the ones of interest - I use one of them in dpdk mode. Thomas>> Did you try to disable CONFIG_RTE_IXGBE_INC_VECTOR? Thanks for the suggestion, I made that change and was giving it some time. Now the result of that is not entirely black and white: previously (in vector mode) my app used to Rx/Tx packets nicely without any hiccups, but would crash in 10 minutes :). Now with this suggested change, its been running for a while and doesnt crash, but the Tx latency and Tx loss is so high (around 10% tx loss) that the app is not doing a great job - but that might just be something that I need to adapt to when using non-vector mode ? I will experiment on that a bit more. So I "think" its fair to say that with the vector disabled, theres no crash, but I need to chase this latency/loss now. Thomas>> Not needed. A DPDK application is fast enough to do the job in 10 minutes ;) Haha, good one :). Thats where I want to get to eventually, but right now some distance from it. Bruce>> Can you perhaps isolate any further the root cause of the issue. For example, does it only occur when you get three packets at the receive ring wraps back around to zero? I will try some more experiments, will read and understand this Rx code a bit more to be able to answer the qn about whether ring wraps around when the problem happens etc.. Rgds, Gopa. On Tue, Jun 30, 2015 at 9:08 AM, Thomas Monjalon wrote: > 2015-06-30 08:49, Gopakumar Choorakkot Edakkunni: >> I am starting to tryout dpdk-2.0.0 with a simple Rx routine very >> similar to the l2fwd example - I am running this on a c3.8xlarge aws >> sr-iov enabled vpc instance (inside the vm it uses ixgbevf driver). > > You mean you are using SR-IOV from Amazon, right? > Do you have more hardware details? > >> Once in every 10 minutes my application crashes in the recieve path. >> And whenever I check the crash reason its because it always has three >> packets in the burst array (I have provided array size of 32) instead >> of the four that it tries to collect in one bunch. And inside >> desc_to_olflags_v(), theres the assumption that there are four >> packets, and obviously it crashes trying to access the fourth buffer. > > Did you try to disable CONFIG_RTE_IXGBE_INC_VECTOR? > >> With a brief look at the code, I really cant make out how its >> guaranteed that we will always have four descriptors fully populated ? >> After the first iteration, the loop does break out if (likely(var != >> RTE_IXGBE_DESCS_PER_LOOP)), but how about the very first iteration >> where we might not have four ? >> >> Any thoughts will be helpful here, trying to get my app working for >> more than 10 minutes :) > > Not needed. A DPDK application is fast enough to do the job in 10 minutes ;) > -- next part -- root at ip-10-0-0-155:/home/ubuntu# cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 62 model name : Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz stepping: 4 microcode : 0x428 cpu MHz : 2793.394 cache size : 25600 KB physical id : 0 siblings: 16 core id : 0 cpu cores : 8 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm xsaveopt fsgsbase smep erms bogomips: 5586.78 clflush size: 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management: processor : 1 vendor_id : GenuineIntel cpu family : 6 model : 62 model name : Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz stepping: 4 microcode : 0x428 cpu MHz : 2793.394 cache size : 25600 KB physical id : 0 siblings: 16 core id : 1 cpu cores : 8 apicid : 2 initial apicid : 2 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology eagerfp
[dpdk-dev] dpdk-2.0.0: crash in ixgbe_recv_scattered_pkts_vec->_recv_raw_pkts_vec->desc_to_olflags_v
Hi, I am starting to tryout dpdk-2.0.0 with a simple Rx routine very similar to the l2fwd example - I am running this on a c3.8xlarge aws sr-iov enabled vpc instance (inside the vm it uses ixgbevf driver). Once in every 10 minutes my application crashes in the recieve path. And whenever I check the crash reason its because it always has three packets in the burst array (I have provided array size of 32) instead of the four that it tries to collect in one bunch. And inside desc_to_olflags_v(), theres the assumption that there are four packets, and obviously it crashes trying to access the fourth buffer. With a brief look at the code, I really cant make out how its guaranteed that we will always have four descriptors fully populated ? After the first iteration, the loop does break out if (likely(var != RTE_IXGBE_DESCS_PER_LOOP)), but how about the very first iteration where we might not have four ? Any thoughts will be helpful here, trying to get my app working for more than 10 minutes :) #0 0x004c8c58 in desc_to_olflags_v (rx_pkts=0x7f1cb0ff17a0, descs=) at /home/gopa/dpdk-2.0.0/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c:173 #1 _recv_raw_pkts_vec (rxq=0x7f1d364fcbc0, rx_pkts=, nb_pkts=, split_packet=0x7f1cb0ff16d0 "") at /home/gopa/dpdk-2.0.0/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c:305 #2 0x004c9cea in ixgbe_recv_scattered_pkts_vec (rx_queue=0x7f1d364fcbc0, rx_pkts=0x7f1cb0ff17a0, nb_pkts=) at /home/gopa/dpdk-2.0.0/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c:475 #3 0x006b9bd8 in rte_eth_rx_burst (port_id=0 '\000', queue_id=0, rx_pkts=0x7f1cb0ff17a0, nb_pkts=32) at /home/gopa/usr/dpdk/include/rte_ethdev.h:2417 --- (gdb) frame 0 #0 0x004c8c58 in desc_to_olflags_v (rx_pkts=0x7f1cb0ff17a0, descs=) at /home/gopa/dpdk-2.0.0/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c:173 173/home/gopa/dpdk-2.0.0/lib/librte_pmd_ixgbe/ixgbe_rxtx_vec.c: No such file or directory. (gdb) p rx_pkts[0] $9 = (struct rte_mbuf *) 0x7f1cf8284640 (gdb) p rx_pkts[1] $10 = (struct rte_mbuf *) 0x7f1d3991061c (gdb) p rx_pkts[2] $11 = (struct rte_mbuf *) 0x7f1d364fc740 (gdb) p rx_pkts[3] $12 = (struct rte_mbuf *) 0x0 (gdb)
