To follow up on this issue, we wanted to recreate this on a simpler host and setup and are able to reproduce this very easily also there. The information and steps are below:
Host - 16 cVPU 8GB host 14.04 Ubuntu LTS running on Xen, single ethernet device OVS - From top of tree https://github/openvswitch/ovs as of this morning (bdc30bfb5c02e1cc6142102423d14184e503487b) Steps to reproduce: Same as before, have traffic coming in and using CT rules, create a bond, add it to bridge, sleep, del-bond, del, bridge, and repeat. Relevant logs from ovs-vswitchd: 2015-12-30T21:39:27.760Z|00097|connmgr|INFO|lan0<->unix: 1 flow_mods in the last 0 s (1 adds) 2015-12-30T21:39:27.763Z|00098|connmgr|INFO|lan0<->unix: 1 flow_mods in the last 0 s (1 adds) 2015-12-30T21:39:57.793Z|00099|ofproto|WARN|lan0: cannot get STP status on nonexistent port 1 2015-12-30T21:39:57.793Z|00100|netlink_notifier|WARN|Dropped 4 log messages in last 30 seconds (most recently, 30 seconds ago) due to excessive rate 2015-12-30T21:39:57.793Z|00101|netlink_notifier|WARN|received bad netlink message 2015-12-30T21:39:57.802Z|00102|bridge|WARN|could not open network device bond0 (No such device) 2015-12-30T21:39:57.880Z|00103|ofproto_dpif|INFO|system@ovs-system: Datapath supports recirculation 2015-12-30T21:39:57.881Z|00104|ofproto_dpif|INFO|system@ovs-system: MPLS label stack length probed as 1 2015-12-30T21:39:57.881Z|00105|ofproto_dpif|INFO|system@ovs-system: Datapath supports unique flow ids 2015-12-30T21:39:57.881Z|00106|ofproto_dpif|INFO|system@ovs-system: Datapath supports ct_state 2015-12-30T21:39:57.881Z|00107|ofproto_dpif|INFO|system@ovs-system: Datapath supports ct_zone 2015-12-30T21:39:57.881Z|00108|ofproto_dpif|INFO|system@ovs-system: Datapath supports ct_mark 2015-12-30T21:39:57.881Z|00109|ofproto_dpif|INFO|system@ovs-system: Datapath supports ct_label 2015-12-30T21:39:57.881Z|00110|ofproto_dpif|INFO|system@ovs-system: Datapath does not support ct_state_nat 2015-12-30T21:39:57.884Z|00001|ofproto_dpif_upcall(handler259)|INFO|received packet on unassociated datapath port 0 2015-12-30T21:39:57.887Z|00111|bridge|INFO|bridge lan0: added interface lan0 on port 65534 2015-12-30T21:39:57.888Z|00112|bridge|INFO|bridge lan0: using datapath ID 0000aabb34438939 2015-12-30T21:39:57.888Z|00113|connmgr|INFO|lan0: added service controller "punix:/usr/local/var/run/openvswitch/lan0.mgmt" 2015-12-30T21:39:57.899Z|00114|bridge|INFO|bridge lan0: added interface bond0 on port 1 2015-12-30T21:39:57.997Z|00115|connmgr|INFO|lan0<->unix: 1 flow_mods in the last 0 s (1 adds) 2015-12-30T21:39:58.000Z|00116|connmgr|INFO|lan0<->unix: 1 flow_mods in the last 0 s (1 adds) 2015-12-30T21:39:58.001Z|00001|dpif(revalidator285)|WARN|system@ovs-system: failed to flow_del (No such file or directory) ufid:e2287aaa-a3b2-4cc2-b8cd-9de2a8fd9926 recirc_id(0x6),dp_hash(0),skb_priority(0),in_port(2),skb_mark(0),ct_state(0x22),ct_zone(0),ct_mark(0),ct_label(0),eth(src=00:16:3e:3a:ce:9e,dst=aa:bb:34:43:89:39),eth_type(0x0800),ipv4(src=10.11.0.54,dst=10.11.13.199,proto=6,tos=0,ttl=64,frag=no),tcp(src=43941,dst=80),tcp_flags(fin|ack) 2015-12-30T21:39:58.001Z|00002|util(revalidator285)|EMER|lib/cmap.c:846: assertion ok failed in cmap_replace() 2015-12-30T21:39:58.043Z|00002|daemon_unix(monitor)|ERR|1 crashes: pid 1686 died, killed (Aborted), core dumped, restarting 2015-12-30T21:39:58.044Z|00003|ovs_numa|INFO|Discovered 16 CPU cores on NUMA node 0 2015-12-30T21:39:58.044Z|00004|ovs_numa|INFO|Discovered 1 NUMA nodes and 16 CPU cores 2015-12-30T21:39:58.044Z|00005|memory|INFO|2496 kB peak resident set size after 174.2 seconds 2015-12-30T21:39:58.045Z|00006|reconnect|INFO|unix:/usr/local/var/run/openvswitch/db.sock: connecting... 2015-12-30T21:39:58.045Z|00007|reconnect|INFO|unix:/usr/local/var/run/openvswitch/db.sock: connected Process state: root@sp-ub1404-s:~# ps aux | grep ovs root 1682 0.0 0.0 9200 2384 ? Ss 13:37 0:00 ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock --remote=db:Open_vSwitch,Open_vSwitch,manager_options --private-key=db:Open_vSwitch,SSL,private_key --certificate=db:Open_vSwitch,SSL,certificate --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --pidfile --detach root 1685 0.0 0.0 13576 1768 ? Ss 13:37 0:00 ovs-vswitchd: monitoring pid 2624 (1 crashes: pid 1686 died, killed (Abort... root 2624 0.1 0.0 1267576 9448 ? Sl 13:39 0:03 ovs-vswitchd --pidfile --detach --monitor --log-file=/var/log/ovs-vswitch.log root 3061 0.0 0.0 11748 2236 pts/1 S+ 14:12 0:00 grep --color=auto ovs root@sp-ub1404-s:~# Backtrace: root@sp-ub1404-s:/var/crash# gdb /usr/local/sbin/ovs-vswitchd /var/crash/core.revalidator285.1686.sp-ub1404-s.1451511598 GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1 Copyright (C) 2014 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html > This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /usr/local/sbin/ovs-vswitchd...done. [New LWP 2586] [New LWP 2589] [New LWP 2578] [New LWP 1840] [New LWP 2464] [New LWP 2587] [New LWP 2577] [New LWP 2580] [New LWP 2590] [New LWP 2582] [New LWP 2588] [New LWP 2583] [New LWP 2584] [New LWP 2576] [New LWP 2581] [New LWP 2579] [New LWP 2574] [New LWP 2585] [New LWP 1686] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `ovs-vswitchd --pidfile --detach --monitor --log-file=/var/log/ovs-vswitch.log'. Program terminated with signal SIGABRT, Aborted. #0 0x00007f8a58982cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 56 ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory. (gdb) bt #0 0x00007f8a58982cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 #1 0x00007f8a589860d8 in __GI_abort () at abort.c:89 #2 0x00000000004e217e in ovs_abort_valist (err_no=err_no@entry=0, format=format@entry=0x54fba0 "%s: assertion %s failed in %s()", args=args@entry=0x7f8a54940cc8) at lib/util.c:323 #3 0x00000000004e8c20 in vlog_abort_valist (module_=<optimized out>, message=0x54fba0 "%s: assertion %s failed in %s()", args=args@entry=0x7f8a54940cc8) at lib/vlog.c:1168 #4 0x00000000004e8ca7 in vlog_abort (module=module@entry=0x7d2160 <VLM_util>, message=message@entry=0x54fba0 "%s: assertion %s failed in %s()") at lib/vlog.c:1182 #5 0x00000000004e1f23 in ovs_assert_failure (where=where@entry=0x52d788 "lib/cmap.c:846", function=function@entry=0x52d7b0 <__func__.6436> "cmap_replace", condition=condition@entry=0x52d785 "ok") at lib/util.c:72 #6 0x0000000000456e60 in cmap_replace (cmap=cmap@entry=0x24f0f20, old_node=old_node@entry=0x7f8a3c008130, new_node=new_node@entry=0x0, hash=3794303658) at lib/cmap.c:846 #7 0x0000000000435452 in cmap_remove (hash=<optimized out>, node=0x7f8a3c008130, cmap=0x24f0f20) at ./lib/cmap.h:265 #8 ukey_delete (ukey=0x7f8a3c008130, umap=0x24f0ef0) at ofproto/ofproto-dpif-upcall.c:1732 #9 push_ukey_ops (udpif=udpif@entry=0x250e610, umap=umap@entry=0x24f0ef0, ops=ops@entry=0x7f8a549412f0, n_ops=n_ops@entry=1) at ofproto/ofproto-dpif-upcall.c:2056 #10 0x00000000004357c8 in revalidator_sweep__ (revalidator=revalidator@entry=0x24be3f0, purge=purge@entry=false) at ofproto/ofproto-dpif-upcall.c:2285 #11 0x0000000000436612 in revalidator_sweep (revalidator=0x24be3f0) at ofproto/ofproto-dpif-upcall.c:2296 #12 udpif_revalidator (arg=0x24be3f0) at ofproto/ofproto-dpif-upcall.c:866 #13 0x00000000004bb9f4 in ovsthread_wrapper (aux_=<optimized out>) at lib/ovs-thread.c:340 #14 0x00007f8a59227182 in start_thread (arg=0x7f8a54943700) at pthread_create.c:312 #15 0x00007f8a58a4647d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111 (gdb) info threads Id Target Id Frame 19 Thread 0x7f8a59653980 (LWP 1686) 0x00007f8a58a3912d in poll () at ../sysdeps/unix/syscall-template.S:81 18 Thread 0x7f8a55144700 (LWP 2585) 0x00007f8a58a3912d in poll () at ../sysdeps/unix/syscall-template.S:81 17 Thread 0x7f8a0ffff700 (LWP 2574) 0x00007f8a58a3912d in poll () at ../sysdeps/unix/syscall-template.S:81 16 Thread 0x7f8a5894b700 (LWP 2579) 0x00007f8a58a3912d in poll () at ../sysdeps/unix/syscall-template.S:81 15 Thread 0x7f8a57949700 (LWP 2581) 0x00007f8a58a3912d in poll () at ../sysdeps/unix/syscall-template.S:81 14 Thread 0x7f8a34ff9700 (LWP 2576) 0x00007f8a58a3912d in poll () at ../sysdeps/unix/syscall-template.S:81 13 Thread 0x7f8a55945700 (LWP 2584) 0x00007f8a58a3912d in poll () at ../sysdeps/unix/syscall-template.S:81 12 Thread 0x7f8a56146700 (LWP 2583) 0x00007f8a58a3912d in poll () at ../sysdeps/unix/syscall-template.S:81 11 Thread 0x7f8a377fe700 (LWP 2588) 0x00007f8a58a3912d in poll () at ../sysdeps/unix/syscall-template.S:81 10 Thread 0x7f8a57148700 (LWP 2582) 0x00007f8a58a3912d in poll () at ../sysdeps/unix/syscall-template.S:81 9 Thread 0x7f8a367fc700 (LWP 2590) 0x00007f8a58a3912d in poll () at ../sysdeps/unix/syscall-template.S:81 8 Thread 0x7f8a5814a700 (LWP 2580) 0x00007f8a58a3912d in poll () at ../sysdeps/unix/syscall-template.S:81 7 Thread 0x7f8a357fa700 (LWP 2577) 0x00007f8a58a3912d in poll () at ../sysdeps/unix/syscall-template.S:81 6 Thread 0x7f8a37fff700 (LWP 2587) 0x00007f8a58a3912d in poll () at ../sysdeps/unix/syscall-template.S:81 5 Thread 0x7f8a59651980 (LWP 2464) pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:238 4 Thread 0x7f8a56947700 (LWP 1840) 0x00007f8a58a3912d in poll () at ../sysdeps/unix/syscall-template.S:81 3 Thread 0x7f8a35ffb700 (LWP 2578) 0x00007f8a58a3912d in poll () at ../sysdeps/unix/syscall-template.S:81 2 Thread 0x7f8a36ffd700 (LWP 2589) 0x00007f8a58a3912d in poll () at ../sysdeps/unix/syscall-template.S:81 * 1 Thread 0x7f8a54943700 (LWP 2586) 0x00007f8a58982cc9 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56 (gdb) This particular crash happened on only the third loop through of me actually starting traffic. I have attached the vswitch log as well from this run. That was fast but it usually happens w/in 5 minutes. The script that is used for creating and manipulating the bridge is below. It allows all TCP traffic from the host out and inbound ssh/web via the new ct_state semantics. All other traffic is allowed. root@sp-ub1404-s:~# cat torture_ovs.sh #/bin/bash -x while : do ix=$((ix+1)) echo "======================" echo "starting iteration $ix" echo "======================" # Destroy it all and cleanup ip link del bond0 ovs-vsctl -- --if-exists del-br lan0 killall dhclient #Recreate the bond and bridge and get an IP ip link add bond0 address aa:bb:34:43:89:39 type bond ip addr flush eth0 ip link set eth0 down master bond0 ip link set bond0 up ovs-vsctl --may-exist add-br lan0 -- set bridge lan0 other-config:hwaddr=aa:bb:34:43:89:39 ovs-vsctl add-port lan0 bond0 dhclient lan0 ifconfig lan0 # Add rules to allow established connection and HTTP/SSH in ovs-ofctl add-flow lan0 "priority=45000,in_port=1,ip,tcp,ct_state=-trk,actions=ct(table=0,zone=0)" ovs-ofctl add-flow lan0 "priority=40000,ip,tcp,ct_state=+trk-new+est,actions=NORMAL" ovs-ofctl add-flow lan0 "priority=35000,in_port=local,ip,tcp,ct_state=+trk+new-est,actions=ct(commit,zone=0),NORMAL" ovs-ofctl add-flow lan0 "priority=35000,in_port=1,ip,tcp,tp_dst=80,ct_state=+trk+new-est,actions=ct(commit,zone=0),NORMAL" ovs-ofctl add-flow lan0 "priority=35000,in_port=1,ip,tcp,tp_dst=22,ct_state=+trk+new-est,actions=ct(commit,zone=0),NORMAL" ovs-ofctl add-flow lan0 "priority=25000,ip,tcp actions=drop" # Wait and start over sleep 30 # Check for crash grep cmap /var/log/ovs-vswitch.log if [[ $? -eq 0 ]]; then date echo "Failed after $ix iterations" exit 1 fi done root@sp-ub1404-s:~# This results in something like this: root@sp-ub1404-s:~# ovs-ofctl dump-flows lan0 NXST_FLOW reply (xid=0x4): cookie=0x0, duration=19.243s, table=0, n_packets=57455, n_bytes=4342507, idle_age=0, priority=45000,ct_state=-trk,tcp,in_port=1 actions=ct(table=0) cookie=0x0, duration=19.241s, table=0, n_packets=90524, n_bytes=95081417, idle_age=0, priority=40000,ct_state=-new+est+trk,tcp actions=NORMAL cookie=0x0, duration=19.239s, table=0, n_packets=0, n_bytes=0, idle_age=19, priority=35000,ct_state=+new-est+trk,tcp,in_port=LOCAL actions=ct(commit),NORMAL cookie=0x0, duration=19.236s, table=0, n_packets=7527, n_bytes=556998, idle_age=1, priority=35000,ct_state=+new-est+trk,tcp,in_port=1,tp_dst=80 actions=ct(commit),NORMAL cookie=0x0, duration=19.234s, table=0, n_packets=0, n_bytes=0, idle_age=19, priority=35000,ct_state=+new-est+trk,tcp,in_port=1,tp_dst=22 actions=ct(commit),NORMAL cookie=0x0, duration=19.232s, table=0, n_packets=0, n_bytes=0, idle_age=19, priority=25000,tcp actions=drop cookie=0x0, duration=19.351s, table=0, n_packets=81, n_bytes=6377, idle_age=0, priority=0 actions=NORMAL root@sp-ub1404-s:~# So we start the script above and then start generating lots of web flows using httperf from an external host directed to this host with another infinite loop that does this: httperf --server=$1 --port=$2 --num-conns=80000 Does this help? Let us know if there is any other information that may be of interest. Again the full log is attached. On Thu, Dec 17, 2015 at 1:48 PM, Ben Warren <b...@skyportsystems.com> wrote: > Hi Joe, > > > On Dec 16, 2015, at 4:19 PM, Joe Stringer <j...@ovn.org> wrote: > > > > Hi Ben, > > > > Thanks for following up on this. Yes I think that all of the patches > > we previously referred to are now merged in some form. > > > > Is the core dump/backtrace the same as before? Could you repost the > > backtrace in this thread? > > > Here’s a current corefile backtrace: > > ----------------- > [Thread debugging using libthread_db enabled] > Using host libthread_db library > "/lib/mips64el-linux-gnuabi64/libthread_db.so.1". > Core was generated by `ovs-vswitchd unix:/var/run/openvswitch/db.sock > -vconsole:emer -vsyslog:err -vfi'. > Program terminated with signal SIGABRT, Aborted. > #0 0x000000eeeaf629dc in raise () from > /lib/mips64el-linux-gnuabi64/libc.so.6 > (gdb) back > #0 0x000000eeeaf629dc in raise () from > /lib/mips64el-linux-gnuabi64/libc.so.6 > #1 0x000000eeeaf64470 in abort () from > /lib/mips64el-linux-gnuabi64/libc.so.6 > #2 0x0000000120152a48 in ovs_abort_valist () > #3 0x000000012015c82c in vlog_abort_valist () > #4 0x000000012015c874 in vlog_abort () > #5 0x00000001201526ac in ovs_assert_failure () > #6 0x00000001200937f8 in cmap_replace () > ----------------- > > And here’s what ovs-vswitchd.log spit out: > > ----------------- > 2015-12-17T21:34:44.170Z|01044|bridge|INFO|bridge lan0: added interface > lan0 on port 65534 > 2015-12-17T21:34:44.171Z|01045|bridge|INFO|bridge lan0: using datapath ID > 0000dc3979807000 > 2015-12-17T21:34:44.172Z|01046|connmgr|INFO|lan0: added service controller > "punix:/var/run/openvswitch/lan0.mgmt" > 2015-12-17T21:34:44.175Z|01047|dpif|WARN|system@ovs-system: failed to > flow_del (No such file or directory) > ufid:b6c466f2-1771-4241-abf6-7ec5297a467b > recirc_id(0),dp_hash(0),skb_priority(0),in_port(7),skb_mark(0x5),ct_state(0x2a),ct_zone(0),ct_mark(0x5),ct_label(0),eth(src=dc:39:79:80:71:60,dst=dc:39:79:80:71:08),eth_type(0x0800),ipv4(src=192.168.27.2,dst=10.0.15.1,proto=6,tos=0,ttl=63,frag=no),tcp(src=25252,dst=50363),tcp_flags(psh|ack) > 2015-12-17T21:34:44.175Z|01048|util|EMER|lib/cmap.c:846: assertion ok > failed in cmap_replace() > 2015-12-17T21:34:44.205Z|00008|daemon_unix(monitor)|ERR|7 crashes: pid > 21768 died, killed (Aborted), restarting > 2015-12-17T21:34:44.215Z|00009|ovs_numa|INFO|Discovered 0 NUMA nodes and 0 > CPU cores > 2015-12-17T21:34:44.215Z|00010|memory|INFO|7584 kB peak resident set size > after 10953.4 seconds > 2015-12-17T21:34:44.215Z|00011|reconnect|INFO|unix:/var/run/openvswitch/db.sock: > connecting... > 2015-12-17T21:34:44.215Z|00012|reconnect|INFO|unix:/var/run/openvswitch/db.sock: > connected > 2015-12-17T21:34:44.422Z|00013|ofproto_dpif|INFO|system@ovs-system: > Datapath supports recirculation > 2015-12-17T21:34:44.423Z|00014|ofproto_dpif|INFO|system@ovs-system: MPLS > label stack length probed as 1 > 2015-12-17T21:34:44.423Z|00015|ofproto_dpif|INFO|system@ovs-system: > Datapath supports unique flow ids > 2015-12-17T21:34:44.423Z|00016|ofproto_dpif|INFO|system@ovs-system: > Datapath supports ct_state > 2015-12-17T21:34:44.423Z|00017|ofproto_dpif|INFO|system@ovs-system: > Datapath supports ct_zone > 2015-12-17T21:34:44.423Z|00018|ofproto_dpif|INFO|system@ovs-system: > Datapath supports ct_mark > 2015-12-17T21:34:44.424Z|00019|ofproto_dpif|INFO|system@ovs-system: > Datapath supports ct_label > 2015-12-17T21:34:44.435Z|00001|ofproto_dpif_upcall(handler1)|INFO|received > packet on unassociated datapath port 0 > ----------------- > > > Can you remind me what the port types are? Are bridges also being > added/removed? > > > > This seems to be the last command issued before the crash (based on > timestamp) > > ----------------- > 2015-12-17T21:34:44.004905Z", "Level": "debug", "Message": "ran [ovs-vsctl > -- --if-exists del-br lan0] > ----------------- > > then this > > ----------------- > 2015-12-17T21:34:44.489405Z", "Level": "debug", "Message": "ran [ovs-vsctl > add-br lan0 -- set bridge lan0 other-config:hwaddr=dc:39:79:80:70:00\n] > ----------------- > > Note: this is on a test bed where we’re continually creating and > destroying the bridge, since it seems to accelerate the problem (we can > make it happen within about 5 minutes). In normal operation, the bridge > stays up and we just delete regular ports. In that case, the problem only > shows itself a couple of times per day at most. > > > Jarno and I briefly spoke about this today, and one thought that came > > up is whether the number of threads makes a difference here. Are you > > also able to reproduce if you, for example, reduce the number of > > revalidator/handler threads to 1? > > > > ovs-vsctl set Open_vSwitch . other_config:n-revalidator-threads=1 > > ovs-vsctl set Open_vSwitch . other_config:n-handler-threads=1 > > > I tried these commands, unfortunately no improvement. > > > > On 16 December 2015 at 11:15, Ben Warren <b...@skyportsystems.com> wrote: > >> Hi, > >> > >> We’re seeing this crash about a couple of times a day on our test bed, > >> always when removing ports from a bridge (As Keith originally > reported). Do > >> you have any idea what might be happening? As I mentioned in my > previous > >> message, we’re pretty well at top of tree, so can very easily test any > >> fixes. > >> > >> thanks, > >> Ben > >> > >> > >> On Dec 8, 2015, at 3:00 PM, Ben Warren <b...@skyportsystems.com> wrote: > >> > >> Hi Joe, > >> > >> Sorry for taking so long to get back to this. > >> > >> On Nov 23, 2015, at 6:54 PM, Joe Stringer <j...@ovn.org> wrote: > >> > >> On 20 November 2015 at 10:05, Keith Holleman <keith.holle...@gmail.com> > >> wrote: > >> > >> > >> Follow-up email here has the backtrace for the second method of > >> reproduction. In this case the bridge is not deleted, it was using the > loop > >> logic of effectively these commands: > >> > >> > >> <snip> > >> > >> Thanks a lot for the report! > >> > >> Would you be able to apply these two patches and see if they fix the > >> issue you are observing? > >> > >> https://patchwork.ozlabs.org/patch/541190/ > >> https://patchwork.ozlabs.org/patch/541191/ > >> > >> > >> Now that your conntrack code has been committed, we decided to build > off the > >> “openvswitch/ovs” repo on Github. I built the top of the “branch-2.5” > >> branch as of this morning: > >> > >> commit: > >> > https://github.com/openvswitch/ovs/commit/2862aeff82a3216ea4592c57299569484cf159ea > >> > >> and still see the crash. The patches listed above do not apply > cleanly: it > >> looks like much (although maybe not all?) of the logic is already > committed. > >> > >> Here’s what I see in /var/log/ovsswitchd.log: > >> > >> 2015-12-08T22:19:38.770Z|01159|bridge|INFO|bridge lan0: using datapath > ID > >> 0000dc39790002b0 > >> 2015-12-08T22:19:38.770Z|01160|connmgr|INFO|lan0: added service > controller > >> "punix:/var/run/openvswitch/lan0.mgmt" > >> 2015-12-08T22:19:38.842Z|01161|dpif|WARN|system@ovs-system: failed to > >> flow_del (No such file or directory) > >> ufid:7580e732-908d-4134-9ca9-f6887195c2ae > >> > recirc_id(0),dp_hash(0),skb_priority(0),in_port(2),skb_mark(0),ct_state(0),ct_zone(0),ct_mark(0),ct_label(0),eth(src=04:00:00:00:00:02,dst=04:00:00:00:00:fe),eth_type(0x0800),ipv4(src=192.168.27.2,dst=192.168.27.254,proto=6,tos=0,ttl=64,frag=no),tcp(src=39055,dst=11111),tcp_flags(psh|ack) > >> 2015-12-08T22:19:38.842Z|01162|util|EMER|lib/cmap.c:846: assertion ok > failed > >> in cmap_replace() > >> 2015-12-08T22:19:39.175Z|00002|daemon_unix(monitor)|ERR|1 crashes: pid > 978 > >> died, killed (Aborted), core dumped, restarting > >> > >> System information: > >> > >> # ovs-ofctl --version > >> ovs-ofctl (Open vSwitch) 2.5.0 > >> Compiled Dec 8 2015 12:16:49 > >> OpenFlow versions 0x1:0x4 > >> > >> # uname -a > >> Linux cd25 3.10.20-rt14-copilot #1 SMP Tue Dec 8 12:11:17 PST 2015 > mips64 > >> GNU/Linux > >> > >> Please let us know what other information we can provide to help figure > this > >> out. > >> > >> regards, > >> Ben > >> > >> _______________________________________________ > >> discuss mailing list > >> discuss@openvswitch.org > >> http://openvswitch.org/mailman/listinfo/discuss > >> > >> > >
ovs-vswitch.log
Description: Binary data
_______________________________________________ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss
