Re: [ovs-discuss] conntrack: Another ct-clean thread crash bug
On 9/8/17, 2:51 AM, "wangyunjian"wrote: The operations of buckets->connections hmap should with a lock to protect between ovs-vswitchd thread and pmd thread(or ct clean thread).But conn_clean() will release the lock. This time, the hmap maybe change by other thread and the node->next maybe remove from hmap. [Darrell] If I read your text correctly, then what you describe is intentional – there are multiple threads and buckets are protected by locks, but, I don’t see a specific issue pointed to. Also, this is not consistent with the stack trace below, assuming the stack trace is generated from non-instrumented code and is reliable. Can you provide initially : 1/ Your config – ovs-vsctl show, sudo ovs-appctl dpif/show, sudo ovs-ofctl dump-flows , ovs-appctl dpif/dump-flows br0 2/ I could not understand your simple diagram – can you make association of the diagram to “ovs-vsctl show” output. 3/ Are you sending TCP traffic or UDP traffic ? 4/ What streams are you sending ? What I mean by streams – what range of src/dst IP addresses and src/dst layer 4 port numbers What physical or virtual ports are you sending the streams from. process_one(){ // pmd thread ct_lock_lock(ctb.lock); conn_clean (){ ct_lock_unlock(ctb.lock); ... ct_lock_lock(ctb.lock) } ct_lock_unlock(ctb.lock); } conntrack_flush() { //main thread ct_lock_lock(ctb.lock); nat_clean () { ct_lock_unlock(ctb.lock); ... ct_lock_lock(ctb.lock) } ct_lock_unlock(ctb.lock); } clean_thread_main() { // ct clean thread ct_lock_lock(ctb.lock); conn_clean (){ ct_lock_unlock(ctb.lock); ... ct_lock_lock(ctb.lock) } ct_lock_unlock(ctb.lock); } > -Original Message- > From: ovs-discuss-boun...@openvswitch.org [mailto:ovs-discuss- > boun...@openvswitch.org] On Behalf Of Darrell Ball > Sent: Wednesday, September 06, 2017 11:50 PM > To: Huanglili (lee) ; ovs- > disc...@openvswitch.org > Cc: b...@nicira.com; caihe ; liucheng (J) > > Subject: Re: [ovs-discuss] conntrack: Another ct-clean thread crash bug > > Hmm, that seems odd. > Also, the code change you propose below does not make sense and would > likely cause similar crashes itself. > > Maybe, you explain what you are trying to do in your testing ? > Can you say what traffic are you sending and from which ports ? > > I’ll take another look at the related code. > > Darrell > > > On 9/6/17, 6:14 AM, "Huanglili (lee)" wrote: > > Hi, > We met another vswitchd crash when we use ct(nat) (ovs+dpdk). > > Program terminated with signal 11, Segmentation fault. > #0 0x00574a0b in hmap_remove (node=0x7f150c6e60a8, > hmap=0x7f1553c40780) at lib/hmap.h:270 > while (*bucket != node) { > > (gdb) bt > #0 0x00574a0b in hmap_remove (node=0x7f150c6e60a8, > hmap=0x7f1553c40780) > #1 sweep_bucket (limit=1808, now=563303851, ctb=0x7f1553c40778, > ct=0x7f1553c3f9a8) > #2 conntrack_clean (now=563303851, ct=0x7f1553c3f9a8) > #3 clean_thread_main (f_=0x7f1553c3f9a8) > > This crash can be triggered by using following flows, maybe the flows are > not reasonable, but shouldn't trigger crash > "table=0,priority=2,in_port=1 actions=resubmit(,2) > table=0,priority=2,in_port=4 actions=resubmit(,2) > table=0,priority=0 actions=drop > table=0,priority=1 actions=resubmit(,10) > table=1,priority=0 actions=resubmit(,14) > table=2,priority=0 actions=resubmit(,4) > table=4,priority=0 actions=resubmit(,14) > table=10,priority=2,arp actions=resubmit(,12) > table=10,priority=1,dl_src=90:E2:BA:69:CD:61 actions=resubmit(,1) > table=10,priority=0 actions=drop > >
Re: [ovs-discuss] How to submit a bug report
I didn't realize that there was different information on the mailing list and in ovs-issues. Thanks for adding the missing information. "git bisect" would probably take 10 tries or less to find the commit, so I'd still like to encourage you to try that approach. On Mon, Sep 11, 2017 at 09:27:46PM +, Llorente Santos Jesus wrote: > Hi Ben, thank you for getting back to me on this. > > I noticed that my previous email with the steps to reproduce was sent to > ovs-dev mailing list. It somewhat felt like the appropriate place to send > such detailed issue. > In any case, I have added those same steps to the open issue in the > ovs-issues repo. I hope that simplifies the follow up! > > I spent quite bit of time looking at the code but I couldn't really see > anything odd. Comparing 2.5.3 with 2.7.0, showed a vast number of commits. I > deepened into the tunnelling code for GRE, VXLAN and Geneve but couldn't stop > the difference. It's quite odd, in my experience, marking packets does not > invalidate in any case receiving them into the socket. > > The issue here is that 2.5.3 is the current LTS version, which in late June > received a new update. The release note says "Bug fixes", so maybe there is > still something we can fix there... > > Best, > Jesus > > > -Original Message- > From: Ben Pfaff [mailto:b...@ovn.org] > Sent: 12 September 2017 00:03 > To: Llorente Santos Jesus> Cc: ovs-discuss@openvswitch.org > Subject: Re: [ovs-discuss] How to submit a bug report > > On Mon, Sep 11, 2017 at 07:27:56PM +, Llorente Santos Jesus wrote: > > I have been seeing buggy behaviour with the tunnelling ports of OvS. I > > would like to ask some help in how to properly submit a bug report I > > created an issue in the git "ovs-issues" > > (https://github.com/openvswitch/ovs-issues/issues/134) about 2 weeks ago, > > but it seems no one is monitoring that. > > I sent an email to b...@openvswitch.org following these instructions > > (http://docs.openvswitch.org/en/latest/internals/bugs/) > > I have also sent an email to this mailing list, but no one has responded my > > message. > > > > Can anyone help me? I would really like to get this annoying bug fixed! > > Your report has definitely been recorded. I can think of a few reasons why > no one has acted on it, besides simply being busy. One is that it does not > give readers a good way to reproduce the problem, even though it talks about > a very particular scenario. Developers are more likely to try to reproduce > and fix a bug when it doesn't require a lot of forethought on how to get > started. Another is that it sounds like the problem has been fixed in a > later version. It's not too rewarding to fix bugs that are already fixed. > > Given that it's been fixed, you could use "git bisect" to try to find out > what commit fixed it. If you reported that, it could allow a developer to > understand and fix the problem without much additional work. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Wireless OVS
On 09/11/17 16:20, S hj wrote: In MANET (mobile ad hoc networks), wireless nodes can communicate together without any access points. Should I still consider hostapd for each wireless node in my simulation? If you aren't running it, probably isn't needed. I mention it only because you had used the word 'wireless'. And there is a patch around for hostapd to talk properly to ovs. I only run OVS on each node; however, it seems that I have to configure something to allow nodes to know about the mac address of each other a quick google for 'manet arp' indicates something like: https://www.ietf.org/mail-archive/web/manet/current/msg08124.html So depending upon the devices you have, the bridging/repeater mode of your node(2) may have issues. Maybe a manet mailing list might have an answer? -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] How to submit a bug report
On Mon, Sep 11, 2017 at 07:27:56PM +, Llorente Santos Jesus wrote: > I have been seeing buggy behaviour with the tunnelling ports of OvS. I would > like to ask some help in how to properly submit a bug report > I created an issue in the git "ovs-issues" > (https://github.com/openvswitch/ovs-issues/issues/134) about 2 weeks ago, but > it seems no one is monitoring that. > I sent an email to b...@openvswitch.org following these instructions > (http://docs.openvswitch.org/en/latest/internals/bugs/) > I have also sent an email to this mailing list, but no one has responded my > message. > > Can anyone help me? I would really like to get this annoying bug fixed! Your report has definitely been recorded. I can think of a few reasons why no one has acted on it, besides simply being busy. One is that it does not give readers a good way to reproduce the problem, even though it talks about a very particular scenario. Developers are more likely to try to reproduce and fix a bug when it doesn't require a lot of forethought on how to get started. Another is that it sounds like the problem has been fixed in a later version. It's not too rewarding to fix bugs that are already fixed. Given that it's been fixed, you could use "git bisect" to try to find out what commit fixed it. If you reported that, it could allow a developer to understand and fix the problem without much additional work. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
[ovs-discuss] How to submit a bug report
I have been seeing buggy behaviour with the tunnelling ports of OvS. I would like to ask some help in how to properly submit a bug report I created an issue in the git "ovs-issues" (https://github.com/openvswitch/ovs-issues/issues/134) about 2 weeks ago, but it seems no one is monitoring that. I sent an email to b...@openvswitch.org following these instructions (http://docs.openvswitch.org/en/latest/internals/bugs/) I have also sent an email to this mailing list, but no one has responded my message. Can anyone help me? I would really like to get this annoying bug fixed! Thank a lot! Jesus ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Wireless OVS
you may want to have a look into how Mininet-WiFi uses OVS in such scenarios. There is code to reproduce a number of MANET use cases, plus an active mailing list (cc:) https://github.com/intrig-unicamp/mininet-wifi/wiki On Mon, Sep 11, 2017 at 1:58 PM, Raymond Burkholderwrote: > >> I am working with CORE Network Emulator and I implemented a MANET including >> three wireless nodes. > > I am not familiar with MANET, but if this is hostapd based? > >> >> I am trying to run OVS on these three nodes. The topology is as follows, >> >> node (1) --- node(2) -- -node(3) >> >> >> In this topology, for the first step, when I run OVS on wireless nodes as a >> normal switch (standalone), node(1) can ping node(2), and node(2) can ping >> node(3). However, node(1) cannot ping node(3). >> >> When I captured the packets, there is no ARP Reply packet to help node (1) >> to know the mac address of node(3). >> (I did not have this problem when I run OVS on the same topology in wired >> networks.) >> >> Could you help me with this issue? Could you let me know how to configure >> OVSs to activate ARP Reply packets and how to see the mac table ? >> >> Thank you >> > > > -- > This message has been scanned for viruses and > dangerous content by MailScanner, and is > believed to be clean. > > ___ > discuss mailing list > disc...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Wireless OVS
In MANET (mobile ad hoc networks), wireless nodes can communicate together without any access points. Should I still consider hostapd for each wireless node in my simulation? I only run OVS on each node; however, it seems that I have to configure something to allow nodes to know about the mac address of each other On Mon, Sep 11, 2017 at 12:58 PM, Raymond Burkholderwrote: > > > I am working with CORE Network Emulator and I implemented a MANET > including three wireless nodes. > > I am not familiar with MANET, but if this is hostapd based? > > > > > I am trying to run OVS on these three nodes. The topology is as follows, > > > > node (1) --- node(2) -- -node(3) > > > > > > In this topology, for the first step, when I run OVS on wireless nodes > as a normal switch (standalone), node(1) can ping node(2), and node(2) can > ping node(3). However, node(1) cannot ping node(3). > > > > When I captured the packets, there is no ARP Reply packet to help node > (1) to know the mac address of node(3). > > (I did not have this problem when I run OVS on the same topology in > wired networks.) > > > > Could you help me with this issue? Could you let me know how to > configure OVSs to activate ARP Reply packets and how to see the mac table ? > > > > Thank you > > > > > -- > This message has been scanned for viruses and > dangerous content by MailScanner, and is > believed to be clean. > > ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
[ovs-discuss] Wireless OVS
Hello, I am working with CORE Network Emulator and I implemented a MANET including three wireless nodes. I am trying to run OVS on these three nodes. The topology is as follows, node (1) --- node(2) -- -node(3) In this topology, for the first step, when I run OVS on wireless nodes as a normal switch (standalone), node(1) can ping node(2), and node(2) can ping node(3). However, node(1) cannot ping node(3). When I captured the packets, there is no ARP Reply packet to help node (1) to know the mac address of node(3). (I did not have this problem when I run OVS on the same topology in wired networks.) Could you help me with this issue? Could you let me know how to configure OVSs to activate ARP Reply packets and how to see the mac table ? Thank you ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] ovsdb-idl.run-msg many update msgs about br-ex and controller
On Mon, Sep 11, 2017 at 04:27:45PM +0800, 顾兆伦 wrote: > Hi all, sorry to bother you. > > When I use python/ovs, idl.run() always costs about 50 ms to execute, and > it runs 50times, what is too long for me. When I check the > msg(msg=self._sesson.recv()), I found this: > [snip almost 2000 lines] > > I want to know why idl.run() recv so many update msgs about br-ex and > controller, it’s because of ‘tx_packets’ changed? How to avoid this so I can > take less time Usually these fields should only update about every 5 seconds. Perhaps some bug is updating them too often. However, if you don't need particular tables or columns, you can simply not monitor those. That is more efficient regardless of how often those tables or columns update. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Compatible DPDK version for 2.4.0 OVS
According to the FAQ, the version is DPDK 2.0: http://docs.openvswitch.org/en/latest/faq/releases/ If that's incorrect, we should update the FAQ. i agree with Ben, though, that's it's probably best to use OVS 2.8. Thanks, --Justin > On Sep 11, 2017, at 7:57 AM, Avi Cohen (A)wrote: > > dpdk-2.1.0. > > From: ovs-discuss-boun...@openvswitch.org > [mailto:ovs-discuss-boun...@openvswitch.org] On Behalf Of Ranjith Kumar D > Sent: Monday, 11 September, 2017 5:48 PM > To: disc...@openvswitch.org > Subject: [ovs-discuss] Compatible DPDK version for 2.4.0 OVS > > Hi, > > I am using DPDK 2.0 for OVS 2.4.0 and DPDK library linking is failing while > configuring OVS. Is there any specific version of DPDK I should use for 2.4.0 > OVS ? > > Regards, > Ranjith > ___ > discuss mailing list > disc...@openvswitch.org > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] How to build openvswitch with Ddpdk enabled debian package (openvswitch-switch-dpdk.2.7.2-1.deb) for OVS ver. 2.7.2
On Mon, Sep 11, 2017 at 11:25:59PM +1200, Brad Cowie wrote: > On 10 September 2017 at 03:17, Ben Pfaffwrote: > > > On Sat, Sep 09, 2017 at 02:26:04AM +, Chou, David J wrote: > > > Hi, > > > > > > I tried to build openvswitch with dpdk enabled debian package > > (openvswitch-switch-dpdk.2.7.2-1.deb) for OVS ver. 2.7.2. > > > > The Debian packaging that comes with OVS doesn't build a DPDK-based > > switch. > > > > The folks at Ubuntu do provide OVS packaging that includes DPDK: > https://git.launchpad.net/~ubuntu-server-dev/ubuntu/+source/openvswitch Oh. I'm surprised they haven't upstreamed them. Wonder why. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] Compatible DPDK version for 2.4.0 OVS
dpdk-2.1.0. From: ovs-discuss-boun...@openvswitch.org [mailto:ovs-discuss-boun...@openvswitch.org] On Behalf Of Ranjith Kumar D Sent: Monday, 11 September, 2017 5:48 PM To: disc...@openvswitch.org Subject: [ovs-discuss] Compatible DPDK version for 2.4.0 OVS Hi, I am using DPDK 2.0 for OVS 2.4.0 and DPDK library linking is failing while configuring OVS. Is there any specific version of DPDK I should use for 2.4.0 OVS ? Regards, Ranjith ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
[ovs-discuss] Compatible DPDK version for 2.4.0 OVS
Hi, I am using DPDK 2.0 for OVS 2.4.0 and DPDK library linking is failing while configuring OVS. Is there any specific version of DPDK I should use for 2.4.0 OVS ? Regards, Ranjith ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] How to build openvswitch with Ddpdk enabled debian package (openvswitch-switch-dpdk.2.7.2-1.deb) for OVS ver. 2.7.2
On 10 September 2017 at 03:17, Ben Pfaffwrote: > On Sat, Sep 09, 2017 at 02:26:04AM +, Chou, David J wrote: > > Hi, > > > > I tried to build openvswitch with dpdk enabled debian package > (openvswitch-switch-dpdk.2.7.2-1.deb) for OVS ver. 2.7.2. > > The Debian packaging that comes with OVS doesn't build a DPDK-based > switch. > The folks at Ubuntu do provide OVS packaging that includes DPDK: https://git.launchpad.net/~ubuntu-server-dev/ubuntu/+source/openvswitch I build and backport these packages to older versions of Ubuntu and Debian and provide them on my mirror: https://packages.wand.net.nz/ Brad ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
Re: [ovs-discuss] [ovs-dev] OVS DPDK NUMA pmd assignment question for physical port
Hi Wang, I believe that the PMD stats processing cycles includes EMC processing time. This is just in the context of your results being surprising. It could be a factor if you are using code where the bug exists. The patch carries a fixes: tag (I think) that should help you figure out if your results were potentially affected by this issue. Regards, /Billy. > -Original Message- > From: 王志克 [mailto:wangzh...@jd.com] > Sent: Monday, September 11, 2017 3:00 AM > To: O Mahony, Billy; ovs- > d...@openvswitch.org; Jan Scheurich ; Darrell > Ball ; ovs-discuss@openvswitch.org; Kevin Traynor > > Subject: RE: [ovs-dev] OVS DPDK NUMA pmd assignment question for > physical port > > Hi Billy, > > In my test, almost all traffic went trough via EMC. So the fix does not impact > the result, especially we want to know the difference (not the exact num). > > Can you test to get some data? Thanks. > > Br, > Wang Zhike > > -Original Message- > From: O Mahony, Billy [mailto:billy.o.mah...@intel.com] > Sent: Friday, September 08, 2017 11:18 PM > To: 王志克; ovs-...@openvswitch.org; Jan Scheurich; Darrell Ball; ovs- > disc...@openvswitch.org; Kevin Traynor > Subject: RE: [ovs-dev] OVS DPDK NUMA pmd assignment question for > physical port > > Hi Wang, > > https://mail.openvswitch.org/pipermail/ovs-dev/2017-August/337309.html > > I see it's been acked and is due to be pushed to master with other changes > on the dpdk merge branch so you'll have to apply it manually for now. > > /Billy. > > > -Original Message- > > From: 王志克 [mailto:wangzh...@jd.com] > > Sent: Friday, September 8, 2017 11:48 AM > > To: ovs-...@openvswitch.org; Jan Scheurich > > ; O Mahony, Billy > > ; Darrell Ball ; ovs- > > disc...@openvswitch.org; Kevin Traynor > > Subject: Re: [ovs-dev] OVS DPDK NUMA pmd assignment question for > > physical port > > > > Hi Billy, > > > > I used ovs2.7.0. I searched the git log, and not sure which commit it > > is. Do you happen to know? > > > > Yes, I cleared the stats after traffic run. > > > > Br, > > Wang Zhike > > > > > > From: "O Mahony, Billy" > > To: "wangzh...@jd.com" , Jan Scheurich > > , Darrell Ball , > > "ovs-discuss@openvswitch.org" , > > "ovs-...@openvswitch.org" , Kevin > Traynor > > > > Subject: Re: [ovs-dev] OVS DPDK NUMA pmd assignment question for > > physical port > > Message-ID: > > <03135aea779d444e90975c2703f148dc58c19...@irsmsx107.ger.c > > orp.intel.com> > > > > Content-Type: text/plain; charset="utf-8" > > > > Hi Wang, > > > > Thanks for the figures. Unexpected results as you say. Two things come > > to > > mind: > > > > I?m not sure what code you are using but the cycles per packet > > statistic was broken for a while recently. Ilya posted a patch to fix > > it so make sure you have that patch included. > > > > Also remember to reset the pmd stats after you start your traffic and > > then measure after a short duration. > > > > Regards, > > Billy. > > > > > > > > From: ??? [mailto:wangzh...@jd.com] > > Sent: Friday, September 8, 2017 8:01 AM > > To: Jan Scheurich ; O Mahony, Billy > > ; Darrell Ball ; ovs- > > disc...@openvswitch.org; ovs-...@openvswitch.org; Kevin Traynor > > > > Subject: RE: [ovs-dev] OVS DPDK NUMA pmd assignment question for > > physical port > > > > > > Hi All, > > > > > > > > I tested below cases, and get some performance data. The data shows > > there is little impact for cross NUMA communication, which is > > different from my expectation. (Previously I mentioned that cross NUMA > > would add 60% cycles, but I can NOT reproduce it any more). > > > > > > > > @Jan, > > > > You mentioned cross NUMA communication would cost lots more cycles. > > Can you share your data? I am not sure whether I made some mistake or > not. > > > > > > > > @All, > > > > Welcome your data if you have data for similar cases. Thanks. > > > > > > > > Case1: VM0->PMD0->NIC0 > > > > Case2:VM1->PMD1->NIC0 > > > > Case3:VM1->PMD0->NIC0 > > > > Case4:NIC0->PMD0->VM0 > > > > Case5:NIC0->PMD1->VM1 > > > > Case6:NIC0->PMD0->VM1 > > > > > > > > ? VM Tx Mpps Host Tx Mpps avg cycles per packet avg processing > > cycles per packet > > > > Case1 1.4 1.4 512 > > 415 > > > > Case2 1.3 1.3 537 > > 436 > > > > Case3 1.351.35 514 390 > > > > > > > > ? VM Rx MppsHost Rx Mpps avg cycles per packet avg processing > cycles > > per
Re: [ovs-discuss] [OVS-Discuss]: configure: error: cannot link with dpdk
Thank you Ben Pfaff, issue is resolved with 2.8.0 OVS. Regards, Ranjith -Original Message- From: Ben Pfaff [mailto:b...@ovn.org] Sent: Saturday, September 09, 2017 10:27 PM To: Ranjith Kumar DCc: disc...@openvswitch.org Subject: Re: [ovs-discuss] [OVS-Discuss]: configure: error: cannot link with dpdk On Fri, Sep 08, 2017 at 01:33:35PM +, Ranjith Kumar D wrote: > Hi, > > I am trying to install OVS with DPDK and I getting below error while > configuring OVS > > configure: error: cannot link with dpdk > > The below are my OVS and DPDK software version details: > > DPDK version : dpdk-stable-17.02.1 > OVS version : rte-openvswitch-2.4.0 It sounds like you're building the Intel-initiated fork of OVS based on DPDK. This isn't the right mailing list to discuss that fork. Also, to the best of my knowledge, that fork has not been under active development for some time. I suggest that you use mainline OVS 2.8.0 (or master) instead of the fork. ___ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss