On 8/6/20 11:54 AM, Brendan Doyle wrote: > I don't see any ovn-northd.log log, I only see those when I'm running > OVN outside the k8s cluster. > Before I start the Satefulset on my k8 nodes I run: > > ovn-ctl stop_northd > ovn-ctl stop_ovsdb > rm -rf /usr/etc/ovn/*.db > > > The only logs I see are ovn-controller.log (I'm running ovn controller > on the K8 nodes) ovsdb-server-nb.log ovsdb-server-sb.log > These logs look normal >
ovn-northd is the daemon that translates OVN_Northbound DB records to OVN_Southbound DB records (including logical flows). If your deployment doesn't start ovn-northd SB won't get populated. Regards, Dumitru > k8 node 1 logs > ============ > 2020-08-05T13:44:51.958Z|00001|vlog|INFO|opened log file > /var/log/ovn/ovsdb-server-nb.log > 2020-08-05T13:44:51.961Z|00002|ovsdb_server|INFO|ovsdb-server (Open > vSwitch) 2.13.90 > 2020-08-05T13:44:51.961Z|00003|reconnect|INFO|tcp:[253.255.0.34]:6643: > connecting... > 2020-08-05T13:44:51.961Z|00004|reconnect|INFO|tcp:[253.255.0.34]:6643: > connected > 2020-08-05T13:44:51.962Z|00005|raft_rpc|INFO|learned cluster ID 6e05 > 2020-08-05T13:44:51.962Z|00006|raft|INFO|tcp:[253.255.0.34]:6643: > learned server ID c8a5 > 2020-08-05T13:44:51.962Z|00007|raft|INFO|server c8a5 is leader for term 1 > 2020-08-05T13:44:51.962Z|00008|raft|INFO|rejecting append_request > because previous entry 1,5 not in local log (mismatch past end of log) > 2020-08-05T13:44:51.964Z|00009|raft|INFO|server c8a5 added to configuration > 2020-08-05T13:44:51.966Z|00010|raft|INFO|server 0d3d added to configuration > 2020-08-05T13:44:51.966Z|00011|raft|INFO|tcp:253.255.0.34:50744: learned > server ID c8a5 > 2020-08-05T13:44:51.966Z|00012|raft|INFO|tcp:253.255.0.34:50744: learned > remote address tcp:[253.255.0.34]:6643 > 2020-08-05T13:44:52.015Z|00013|raft|INFO|tcp:253.255.0.35:51780: learned > server ID 941e > 2020-08-05T13:44:52.015Z|00014|raft|INFO|tcp:253.255.0.35:51780: learned > remote address tcp:[253.255.0.35]:6643 > 2020-08-05T13:44:52.015Z|00015|raft|INFO|adding 941e (941e at > tcp:[253.255.0.35]:6643) to cluster 6e05 failed (not leader) > 2020-08-05T13:44:52.015Z|00016|raft|INFO|server 941e added to configuration > 2020-08-05T13:44:52.016Z|00017|reconnect|INFO|tcp:[253.255.0.35]:6643: > connecting... > 2020-08-05T13:44:52.016Z|00018|reconnect|INFO|tcp:[253.255.0.35]:6643: > connected > 2020-08-05T13:45:01.962Z|00019|memory|INFO|8424 kB peak resident set > size after 10.0 seconds > 2020-08-05T13:45:01.962Z|00020|memory|INFO|cells:40 monitors:0 > raft-connections:4 > 2020-08-05T14:10:03.593Z|00021|reconnect|INFO|tcp:[253.255.0.34]:6643: > connection closed by peer > 2020-08-05T14:10:03.595Z|00022|raft|INFO|server 941e is leader for term 2 > 2020-08-05T14:10:04.594Z|00023|reconnect|INFO|tcp:[253.255.0.34]:6643: > connecting... > 2020-08-05T14:10:04.594Z|00024|reconnect|INFO|tcp:[253.255.0.34]:6643: > connection attempt failed (Connection refused) > 2020-08-05T14:10:04.594Z|00025|reconnect|INFO|tcp:[253.255.0.34]:6643: > waiting 2 seconds before reconnect > 2020-08-05T14:10:06.594Z|00026|reconnect|INFO|tcp:[253.255.0.34]:6643: > connecting... > 2020-08-05T14:10:06.594Z|00027|reconnect|INFO|tcp:[253.255.0.34]:6643: > connection attempt failed (Connection refused) > 2020-08-05T14:10:06.594Z|00028|reconnect|INFO|tcp:[253.255.0.34]:6643: > waiting 4 seconds before reconnect > 2020-08-05T14:10:08.453Z|00029|raft|INFO|received leadership transfer > from 941e in term 2 > 2020-08-05T14:10:08.453Z|00030|raft|INFO|term 3: starting election > 2020-08-05T14:10:08.453Z|00031|reconnect|INFO|tcp:[253.255.0.35]:6643: > connection closed by peer > > 2020-08-05T13:44:48.971Z|00001|vlog|INFO|opened log file > /var/log/ovn/ovsdb-server-sb.log > 2020-08-05T13:44:48.973Z|00002|ovsdb_server|INFO|ovsdb-server (Open > vSwitch) 2.13.90 > 2020-08-05T13:44:48.973Z|00003|reconnect|INFO|tcp:[253.255.0.34]:6644: > connecting... > 2020-08-05T13:44:48.973Z|00004|reconnect|INFO|tcp:[253.255.0.34]:6644: > connected > 2020-08-05T13:44:48.973Z|00005|raft_rpc|INFO|learned cluster ID 281c > 2020-08-05T13:44:48.973Z|00006|raft|INFO|tcp:[253.255.0.34]:6644: > learned server ID eeed > 2020-08-05T13:44:48.974Z|00007|raft|INFO|server eeed is leader for term 1 > 2020-08-05T13:44:48.974Z|00008|raft|INFO|rejecting append_request > because previous entry 1,4 not in local log (mismatch past end of log) > 2020-08-05T13:44:48.976Z|00009|raft|INFO|server eeed added to configuration > 2020-08-05T13:44:48.977Z|00010|raft|INFO|server 5098 added to configuration > 2020-08-05T13:44:48.977Z|00011|raft|INFO|tcp:253.255.0.34:50628: learned > server ID eeed > 2020-08-05T13:44:48.977Z|00012|raft|INFO|tcp:253.255.0.34:50628: learned > remote address tcp:[253.255.0.34]:6644 > 2020-08-05T13:44:49.044Z|00013|raft|INFO|tcp:253.255.0.35:49594: learned > server ID b9de > 2020-08-05T13:44:49.044Z|00014|raft|INFO|tcp:253.255.0.35:49594: learned > remote address tcp:[253.255.0.35]:6644 > 2020-08-05T13:44:49.044Z|00015|raft|INFO|adding b9de (b9de at > tcp:[253.255.0.35]:6644) to cluster 281c failed (not leader) > 2020-08-05T13:44:49.044Z|00016|raft|INFO|server b9de added to configuration > 2020-08-05T13:44:49.044Z|00017|reconnect|INFO|tcp:[253.255.0.35]:6644: > connecting... > 2020-08-05T13:44:49.045Z|00018|reconnect|INFO|tcp:[253.255.0.35]:6644: > connected > 2020-08-05T13:44:58.975Z|00019|memory|INFO|8408 kB peak resident set > size after 10.0 seconds > 2020-08-05T13:44:58.975Z|00020|memory|INFO|cells:39 monitors:0 > raft-connections:4 > 2020-08-05T14:10:03.593Z|00021|reconnect|INFO|tcp:[253.255.0.34]:6644: > connection closed by peer > 2020-08-05T14:10:03.595Z|00022|raft|INFO|server b9de is leader for term 2 > 2020-08-05T14:10:04.595Z|00023|reconnect|INFO|tcp:[253.255.0.34]:6644: > connecting... > 2020-08-05T14:10:04.595Z|00024|reconnect|INFO|tcp:[253.255.0.34]:6644: > connection attempt failed (Connection refused) > 2020-08-05T14:10:04.595Z|00025|reconnect|INFO|tcp:[253.255.0.34]:6644: > waiting 2 seconds before reconnect > > 2020-08-05T13:47:27.231Z|00001|vlog|INFO|opened log file > /usr/var/log/ovn/ovn-controller.log > 2020-08-05T13:47:27.233Z|00002|reconnect|INFO|unix:/usr/var/run/openvswitch/db.sock: > connecting... > 2020-08-05T13:47:27.233Z|00003|reconnect|INFO|unix:/usr/var/run/openvswitch/db.sock: > connected > 2020-08-05T13:47:27.235Z|00004|main|INFO|OVS IDL reconnected, force > recompute. > 2020-08-05T13:47:27.236Z|00005|reconnect|INFO|tcp:253.255.0.35:6642: > connecting... > 2020-08-05T13:47:27.236Z|00006|main|INFO|OVNSB IDL reconnected, force > recompute. > 2020-08-05T13:47:27.236Z|00007|reconnect|INFO|tcp:253.255.0.35:6642: > connected > 2020-08-05T13:47:27.237Z|00008|ofctrl|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: > connecting to switch > 2020-08-05T13:47:27.237Z|00009|rconn|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: > connecting... > 2020-08-05T13:47:27.237Z|00010|rconn|WARN|unix:/usr/var/run/openvswitch/br-int.mgmt: > connection failed (No such file or directory) > 2020-08-05T13:47:27.237Z|00011|rconn|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: > waiting 1 seconds before reconnect > 2020-08-05T13:47:28.239Z|00012|rconn|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: > connecting... > 2020-08-05T13:47:28.239Z|00013|rconn|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: > connected > 2020-08-05T13:47:28.240Z|00001|pinctrl(ovn_pinctrl0)|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: > connecting to switch > 2020-08-05T13:47:28.240Z|00002|rconn(ovn_pinctrl0)|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: > connecting... > 2020-08-05T13:47:28.240Z|00003|rconn(ovn_pinctrl0)|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: > connected > 2020-08-05T13:47:28.436Z|00014|dpif_netlink|INFO|The kernel module does > not support meters. > 2020-08-05T14:10:05.486Z|00015|main|INFO|OVNSB commit failed, force > recompute next time. > 2020-08-05T14:10:05.486Z|00016|jsonrpc|WARN|tcp:253.255.0.35:6642: > receive error: Connection reset by peer > 2020-08-05T14:10:05.486Z|00017|reconnect|WARN|tcp:253.255.0.35:6642: > connection dropped (Connection reset by peer) > 2020-08-05T14:10:05.486Z|00018|reconnect|INFO|tcp:253.255.0.33:6642: > connecting... > 2020-08-05T14:10:05.486Z|00019|reconnect|INFO|tcp:253.255.0.33:6642: > connection attempt failed (Connection refused) > 2020-08-05T14:10:05.487Z|00020|reconnect|INFO|tcp:253.255.0.34:6642: > connecting... > 2020-08-05T14:10:05.487Z|00021|reconnect|INFO|tcp:253.255.0.34:6642: > connection attempt failed (Connection refused) > 2020-08-05T14:10:06.488Z|00022|reconnect|INFO|tcp:253.255.0.35:6642: > connecting... > 2020-08-05T14:10:06.488Z|00023|reconnect|INFO|tcp:253.255.0.35:6642: > connection attempt failed (Connection refused) > 2020-08-05T14:10:06.488Z|00024|reconnect|INFO|tcp:253.255.0.35:6642: > waiting 2 seconds before reconnect > 2020-08-05T14:10:08.490Z|00025|reconnect|INFO|tcp:253.255.0.33:6642: > connecting... > 2020-08-05T14:10:08.490Z|00026|reconnect|INFO|tcp:253.255.0.33:6642: > connection attempt failed (Connection refused) > 2020-08-05T14:10:08.490Z|00027|reconnect|INFO|tcp:253.255.0.33:6642: > waiting 4 seconds before reconnect > 2020-08-05T14:10:12.492Z|00028|reconnect|INFO|tcp:253.255.0.34:6642: > connecting... > 2020-08-05T14:10:12.492Z|00029|reconnect|INFO|tcp:253.255.0.34:6642: > connection attempt failed (Connection refused) > 2020-08-05T14:10:12.492Z|00030|reconnect|INFO|tcp:253.255.0.34:6642: > continuing to reconnect in the background but suppressing further logging > > K8 node 2 > --------------- > 2020-08-05T13:43:11.094Z|00001|vlog|INFO|opened log file > /var/log/ovn/ovsdb-server-nb.log > 2020-08-05T13:43:11.095Z|00002|raft|INFO|term 1: 40130328487 ms timeout > expired, starting election > 2020-08-05T13:43:11.095Z|00003|raft|INFO|term 1: elected leader by 1+ of > 1 servers > 2020-08-05T13:43:11.098Z|00004|ovsdb_server|INFO|ovsdb-server (Open > vSwitch) 2.13.90 > 2020-08-05T13:43:12.074Z|00005|jsonrpc|WARN|unix#4: send error: Broken pipe > 2020-08-05T13:43:12.074Z|00006|reconnect|WARN|unix#4: connection dropped > (Broken pipe) > 2020-08-05T13:43:15.942Z|00007|jsonrpc|WARN|tcp:253.255.0.35:60448: > receive error: Connection reset by peer > 2020-08-05T13:43:15.942Z|00008|reconnect|WARN|tcp:253.255.0.35:60448: > connection dropped (Connection reset by peer) > 2020-08-05T13:43:15.947Z|00009|raft|INFO|tcp:253.255.0.33:46572: learned > server ID 0d3d > 2020-08-05T13:43:15.947Z|00010|raft|INFO|tcp:253.255.0.33:46572: learned > remote address tcp:[253.255.0.33]:6643 > 2020-08-05T13:43:15.947Z|00011|raft|INFO|starting to add server 0d3d > (0d3d at tcp:[253.255.0.33]:6643) to cluster 6e05 > 2020-08-05T13:43:15.950Z|00012|raft|INFO|cluster 6e05: installed > snapshot on server 0d3d up to 1:1 > 2020-08-05T13:43:15.951Z|00013|reconnect|INFO|tcp:[253.255.0.33]:6643: > connecting... > 2020-08-05T13:43:15.952Z|00014|reconnect|INFO|tcp:[253.255.0.33]:6643: > connected > 2020-08-05T13:43:15.952Z|00015|raft|INFO|adding 0d3d (0d3d at > tcp:[253.255.0.33]:6643) to cluster 6e05 succeeded (completed) > 2020-08-05T13:43:15.997Z|00016|raft|INFO|tcp:253.255.0.35:60500: learned > server ID 941e > 2020-08-05T13:43:15.997Z|00017|raft|INFO|tcp:253.255.0.35:60500: learned > remote address tcp:[253.255.0.35]:6643 > 2020-08-05T13:43:15.997Z|00018|raft|INFO|starting to add server 941e > (941e at tcp:[253.255.0.35]:6643) to cluster 6e05 > 2020-08-05T13:43:16.000Z|00019|raft|INFO|cluster 6e05: installed > snapshot on server 941e up to 1:1 > 2020-08-05T13:43:16.001Z|00020|reconnect|INFO|tcp:[253.255.0.35]:6643: > connecting... > 2020-08-05T13:43:16.001Z|00021|reconnect|INFO|tcp:[253.255.0.35]:6643: > connected > 2020-08-05T13:43:16.002Z|00022|raft|INFO|adding 941e (941e at > tcp:[253.255.0.35]:6643) to cluster 6e05 succeeded (completed) > 2020-08-05T13:43:21.101Z|00023|memory|INFO|8452 kB peak resident set > size after 10.0 seconds > 2020-08-05T13:43:21.101Z|00024|memory|INFO|cells:40 monitors:0 > raft-connections:4 > > 2020-08-05T13:43:11.105Z|00001|vlog|INFO|opened log file > /var/log/ovn/ovsdb-server-sb.log > 2020-08-05T13:43:11.106Z|00002|raft|INFO|term 1: 40130328497 ms timeout > expired, starting election > 2020-08-05T13:43:11.106Z|00003|raft|INFO|term 1: elected leader by 1+ of > 1 servers > 2020-08-05T13:43:11.108Z|00004|ovsdb_server|INFO|ovsdb-server (Open > vSwitch) 2.13.90 > 2020-08-05T13:43:12.959Z|00005|raft|INFO|tcp:253.255.0.33:46140: learned > server ID 5098 > 2020-08-05T13:43:12.959Z|00006|raft|INFO|tcp:253.255.0.33:46140: learned > remote address tcp:[253.255.0.33]:6644 > 2020-08-05T13:43:12.959Z|00007|raft|INFO|starting to add server 5098 > (5098 at tcp:[253.255.0.33]:6644) to cluster 281c > 2020-08-05T13:43:12.962Z|00008|raft|INFO|cluster 281c: installed > snapshot on server 5098 up to 1:1 > 2020-08-05T13:43:12.963Z|00009|reconnect|INFO|tcp:[253.255.0.33]:6644: > connecting... > 2020-08-05T13:43:12.963Z|00010|reconnect|INFO|tcp:[253.255.0.33]:6644: > connected > 2020-08-05T13:43:12.963Z|00011|raft|INFO|adding 5098 (5098 at > tcp:[253.255.0.33]:6644) to cluster 281c succeeded (completed) > 2020-08-05T13:43:12.972Z|00012|jsonrpc|WARN|tcp:253.255.0.35:40706: > receive error: Connection reset by peer > 2020-08-05T13:43:12.972Z|00013|reconnect|WARN|tcp:253.255.0.35:40706: > connection dropped (Connection reset by peer) > 2020-08-05T13:43:13.027Z|00014|raft|INFO|tcp:253.255.0.35:44172: learned > server ID b9de > 2020-08-05T13:43:13.027Z|00015|raft|INFO|tcp:253.255.0.35:44172: learned > remote address tcp:[253.255.0.35]:6644 > 2020-08-05T13:43:13.027Z|00016|raft|INFO|starting to add server b9de > (b9de at tcp:[253.255.0.35]:6644) to cluster 281c > 2020-08-05T13:43:13.029Z|00017|raft|INFO|cluster 281c: installed > snapshot on server b9de up to 1:1 > 2020-08-05T13:43:13.030Z|00018|reconnect|INFO|tcp:[253.255.0.35]:6644: > connecting... > 2020-08-05T13:43:13.030Z|00019|reconnect|INFO|tcp:[253.255.0.35]:6644: > connected > 2020-08-05T13:43:13.031Z|00020|raft|INFO|adding b9de (b9de at > tcp:[253.255.0.35]:6644) to cluster 281c succeeded (completed) > 2020-08-05T13:43:21.109Z|00021|memory|INFO|8604 kB peak resident set > size after 10.0 seconds > 2020-08-05T13:43:21.109Z|00022|memory|INFO|cells:39 monitors:0 > raft-connections:4 > > 2020-08-05T13:45:52.338Z|00001|vlog|INFO|opened log file > /usr/var/log/ovn/ovn-controller.log > 2020-08-05T13:45:52.340Z|00002|reconnect|INFO|unix:/usr/var/run/openvswitch/db.sock: > connecting... > 2020-08-05T13:45:52.340Z|00003|reconnect|INFO|unix:/usr/var/run/openvswitch/db.sock: > connected > 2020-08-05T13:45:52.342Z|00004|main|INFO|OVS IDL reconnected, force > recompute. > 2020-08-05T13:45:52.342Z|00005|reconnect|INFO|tcp:253.255.0.35:6642: > connecting... > 2020-08-05T13:45:52.342Z|00006|main|INFO|OVNSB IDL reconnected, force > recompute. > 2020-08-05T13:45:52.342Z|00007|reconnect|INFO|tcp:253.255.0.35:6642: > connected > 2020-08-05T13:45:52.344Z|00008|ofctrl|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: > connecting to switch > 2020-08-05T13:45:52.344Z|00009|rconn|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: > connecting... > 2020-08-05T13:45:52.344Z|00010|rconn|WARN|unix:/usr/var/run/openvswitch/br-int.mgmt: > connection failed (No such file or directory) > 2020-08-05T13:45:52.344Z|00011|rconn|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: > waiting 1 seconds before reconnect > 2020-08-05T13:45:53.344Z|00012|rconn|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: > connecting... > 2020-08-05T13:45:53.345Z|00013|rconn|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: > connected > 2020-08-05T13:45:53.346Z|00014|dpif_netlink|INFO|The kernel module does > not support meters. > 2020-08-05T13:45:53.348Z|00001|pinctrl(ovn_pinctrl0)|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: > connecting to switch > 2020-08-05T13:45:53.348Z|00002|rconn(ovn_pinctrl0)|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: > connecting... > 2020-08-05T13:45:53.348Z|00003|rconn(ovn_pinctrl0)|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: > connected > 2020-08-05T14:08:29.432Z|00015|main|INFO|OVNSB commit failed, force > recompute next time. > 2020-08-05T14:08:29.432Z|00016|jsonrpc|WARN|tcp:253.255.0.35:6642: > receive error: Connection reset by peer > 2020-08-05T14:08:29.432Z|00017|reconnect|WARN|tcp:253.255.0.35:6642: > connection dropped (Connection reset by peer) > 2020-08-05T14:08:29.432Z|00018|reconnect|INFO|tcp:253.255.0.33:6642: > connecting... > 2020-08-05T14:08:29.432Z|00019|reconnect|INFO|tcp:253.255.0.33:6642: > connection attempt failed (Connection refused) > 2020-08-05T14:08:29.433Z|00020|reconnect|INFO|tcp:253.255.0.34:6642: > connecting... > 2020-08-05T14:08:29.433Z|00021|reconnect|INFO|tcp:253.255.0.34:6642: > connection attempt failed (Connection refused) > 2020-08-05T14:08:30.434Z|00022|reconnect|INFO|tcp:253.255.0.35:6642: > connecting... > 2020-08-05T14:08:30.434Z|00023|reconnect|INFO|tcp:253.255.0.35:6642: > connection attempt failed (Connection refused) > 2020-08-05T14:08:30.434Z|00024|reconnect|INFO|tcp:253.255.0.35:6642: > waiting 2 seconds before reconnect > 2020-08-05T14:08:32.436Z|00025|reconnect|INFO|tcp:253.255.0.33:6642: > connecting... > 2020-08-05T14:08:32.436Z|00026|reconnect|INFO|tcp:253.255.0.33:6642: > connection attempt failed (Connection refused) > 2020-08-05T14:08:32.436Z|00027|reconnect|INFO|tcp:253.255.0.33:6642: > waiting 4 seconds before reconnect > 2020-08-05T14:08:36.439Z|00028|reconnect|INFO|tcp:253.255.0.34:6642: > connecting... > 2020-08-05T14:08:36.439Z|00029|reconnect|INFO|tcp:253.255.0.34:6642: > connection attempt failed (Connection refused) > 2020-08-05T14:08:36.439Z|00030|reconnect|INFO|tcp:253.255.0.34:6642: > continuing to reconnect in the background but suppressing further logging > > k8 node 3 > -------------- > 2020-08-05T13:44:52.007Z|00001|vlog|INFO|opened log file > /var/log/ovn/ovsdb-server-nb.log > 2020-08-05T13:44:52.009Z|00002|ovsdb_server|INFO|ovsdb-server (Open > vSwitch) 2.13.90 > 2020-08-05T13:44:52.010Z|00003|reconnect|INFO|tcp:[253.255.0.34]:6643: > connecting... > 2020-08-05T13:44:52.010Z|00004|reconnect|INFO|tcp:[253.255.0.34]:6643: > connected > 2020-08-05T13:44:52.010Z|00005|raft_rpc|INFO|learned cluster ID 6e05 > 2020-08-05T13:44:52.010Z|00006|raft|INFO|tcp:[253.255.0.34]:6643: > learned server ID c8a5 > 2020-08-05T13:44:52.011Z|00007|raft|INFO|server c8a5 is leader for term 1 > 2020-08-05T13:44:52.011Z|00008|raft|INFO|rejecting append_request > because previous entry 1,6 not in local log (mismatch past end of log) > 2020-08-05T13:44:52.013Z|00009|raft|INFO|server c8a5 added to configuration > 2020-08-05T13:44:52.014Z|00010|raft|INFO|server 0d3d added to configuration > 2020-08-05T13:44:52.014Z|00011|reconnect|INFO|tcp:[253.255.0.33]:6643: > connecting... > 2020-08-05T13:44:52.014Z|00012|reconnect|INFO|tcp:[253.255.0.33]:6643: > connected > 2020-08-05T13:44:52.015Z|00013|raft|INFO|server 941e added to configuration > 2020-08-05T13:44:52.015Z|00014|raft|INFO|tcp:253.255.0.34:52448: learned > server ID c8a5 > 2020-08-05T13:44:52.015Z|00015|raft|INFO|tcp:253.255.0.34:52448: learned > remote address tcp:[253.255.0.34]:6643 > 2020-08-05T13:44:52.015Z|00016|raft|INFO|tcp:253.255.0.33:56762: learned > server ID 0d3d > 2020-08-05T13:44:52.015Z|00017|raft|INFO|tcp:253.255.0.33:56762: learned > remote address tcp:[253.255.0.33]:6643 > 2020-08-05T13:45:02.010Z|00018|memory|INFO|8884 kB peak resident set > size after 10.0 seconds > 2020-08-05T13:45:02.010Z|00019|memory|INFO|cells:40 monitors:0 > raft-connections:4 > 2020-08-05T14:10:03.593Z|00020|raft|INFO|received leadership transfer > from c8a5 in term 1 > 2020-08-05T14:10:03.593Z|00021|raft|INFO|term 2: starting election > 2020-08-05T14:10:03.593Z|00022|reconnect|INFO|tcp:[253.255.0.34]:6643: > connection closed by peer > 2020-08-05T14:10:03.594Z|00023|raft|INFO|term 2: elected leader by 2+ of > 3 servers > 2020-08-05T14:10:04.594Z|00024|reconnect|INFO|tcp:[253.255.0.34]:6643: > connecting... > 2020-08-05T14:10:04.594Z|00025|reconnect|INFO|tcp:[253.255.0.34]:6643: > connection attempt failed (Connection refused) > 2020-08-05T14:10:04.594Z|00026|reconnect|INFO|tcp:[253.255.0.34]:6643: > waiting 2 seconds before reconnect > 2020-08-05T14:10:06.595Z|00027|reconnect|INFO|tcp:[253.255.0.34]:6643: > connecting... > 2020-08-05T14:10:06.595Z|00028|reconnect|INFO|tcp:[253.255.0.34]:6643: > connection attempt failed (Connection refused) > 2020-08-05T14:10:06.595Z|00029|reconnect|INFO|tcp:[253.255.0.34]:6643: > waiting 4 seconds before reconnect > > > cat /usr/var/log/ovn//backups/ovsdb-server-sb.13 > > 2020-08-05T13:44:49.037Z|00001|vlog|INFO|opened log file > /var/log/ovn/ovsdb-server-sb.log > 2020-08-05T13:44:49.039Z|00002|ovsdb_server|INFO|ovsdb-server (Open > vSwitch) 2.13.90 > 2020-08-05T13:44:49.040Z|00003|reconnect|INFO|tcp:[253.255.0.34]:6644: > connecting... > 2020-08-05T13:44:49.040Z|00004|reconnect|INFO|tcp:[253.255.0.34]:6644: > connected > 2020-08-05T13:44:49.040Z|00005|raft_rpc|INFO|learned cluster ID 281c > 2020-08-05T13:44:49.040Z|00006|raft|INFO|tcp:[253.255.0.34]:6644: > learned server ID eeed > 2020-08-05T13:44:49.040Z|00007|raft|INFO|server eeed is leader for term 1 > 2020-08-05T13:44:49.040Z|00008|raft|INFO|rejecting append_request > because previous entry 1,5 not in local log (mismatch past end of log) > 2020-08-05T13:44:49.043Z|00009|raft|INFO|server eeed added to configuration > 2020-08-05T13:44:49.043Z|00010|raft|INFO|server 5098 added to configuration > 2020-08-05T13:44:49.043Z|00011|reconnect|INFO|tcp:[253.255.0.33]:6644: > connecting... > 2020-08-05T13:44:49.043Z|00012|reconnect|INFO|tcp:[253.255.0.33]:6644: > connected > 2020-08-05T13:44:49.044Z|00013|raft|INFO|server b9de added to configuration > 2020-08-05T13:44:49.044Z|00014|raft|INFO|tcp:253.255.0.34:33550: learned > server ID eeed > 2020-08-05T13:44:49.044Z|00015|raft|INFO|tcp:253.255.0.34:33550: learned > remote address tcp:[253.255.0.34]:6644 > 2020-08-05T13:44:49.044Z|00016|raft|INFO|tcp:253.255.0.33:58002: learned > server ID 5098 > 2020-08-05T13:44:49.044Z|00017|raft|INFO|tcp:253.255.0.33:58002: learned > remote address tcp:[253.255.0.33]:6644 > 2020-08-05T13:44:59.041Z|00018|memory|INFO|8652 kB peak resident set > size after 10.0 seconds > 2020-08-05T13:44:59.041Z|00019|memory|INFO|cells:39 monitors:0 > raft-connections:4 > 2020-08-05T14:10:03.593Z|00020|raft|INFO|received leadership transfer > from eeed in term 1 > 2020-08-05T14:10:03.593Z|00021|raft|INFO|term 2: starting election > 2020-08-05T14:10:03.593Z|00022|reconnect|INFO|tcp:[253.255.0.34]:6644: > connection closed by peer > 2020-08-05T14:10:03.594Z|00023|raft|INFO|term 2: elected leader by 2+ of > 3 servers > 2020-08-05T14:10:04.594Z|00024|reconnect|INFO|tcp:[253.255.0.34]:6644: > connecting... > 2020-08-05T14:10:04.594Z|00025|reconnect|INFO|tcp:[253.255.0.34]:6644: > connection attempt failed (Connection refused) > 2020-08-05T14:10:04.594Z|00026|reconnect|INFO|tcp:[253.255.0.34]:6644: > waiting 2 seconds before reconnect > 2020-08-05T14:10:05.378Z|00027|reconnect|INFO|tcp:[253.255.0.33]:6644: > connection closed by peer > > > 2020-08-05T13:47:29.329Z|00001|vlog|INFO|opened log file > /usr/var/log/ovn/ovn-controller.log > 2020-08-05T13:47:29.331Z|00002|reconnect|INFO|unix:/usr/var/run/openvswitch/db.sock: > connecting... > 2020-08-05T13:47:29.331Z|00003|reconnect|INFO|unix:/usr/var/run/openvswitch/db.sock: > connected > 2020-08-05T13:47:29.334Z|00004|main|INFO|OVS IDL reconnected, force > recompute. > 2020-08-05T13:47:29.334Z|00005|reconnect|INFO|tcp:253.255.0.34:6642: > connecting... > 2020-08-05T13:47:29.334Z|00006|main|INFO|OVNSB IDL reconnected, force > recompute. > 2020-08-05T13:47:29.334Z|00007|reconnect|INFO|tcp:253.255.0.34:6642: > connected > 2020-08-05T13:47:29.336Z|00008|ofctrl|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: > connecting to switch > 2020-08-05T13:47:29.336Z|00009|rconn|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: > connecting... > 2020-08-05T13:47:29.336Z|00010|rconn|WARN|unix:/usr/var/run/openvswitch/br-int.mgmt: > connection failed (No such file or directory) > 2020-08-05T13:47:29.336Z|00011|rconn|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: > waiting 1 seconds before reconnect > 2020-08-05T13:47:30.337Z|00012|rconn|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: > connecting... > 2020-08-05T13:47:30.338Z|00013|rconn|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: > connected > 2020-08-05T13:47:30.340Z|00014|dpif_netlink|INFO|The kernel module does > not support meters. > 2020-08-05T13:47:30.341Z|00001|pinctrl(ovn_pinctrl0)|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: > connecting to switch > 2020-08-05T13:47:30.341Z|00002|rconn(ovn_pinctrl0)|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: > connecting... > 2020-08-05T13:47:30.341Z|00003|rconn(ovn_pinctrl0)|INFO|unix:/usr/var/run/openvswitch/br-int.mgmt: > connected > 2020-08-05T14:10:03.592Z|00015|main|INFO|OVNSB commit failed, force > recompute next time. > 2020-08-05T14:10:03.593Z|00016|jsonrpc|WARN|tcp:253.255.0.34:6642: > receive error: Connection reset by peer > 2020-08-05T14:10:03.593Z|00017|reconnect|WARN|tcp:253.255.0.34:6642: > connection dropped (Connection reset by peer) > 2020-08-05T14:10:03.593Z|00018|reconnect|INFO|tcp:253.255.0.33:6642: > connecting... > 2020-08-05T14:10:03.593Z|00019|reconnect|INFO|tcp:253.255.0.33:6642: > connected > 2020-08-05T14:10:05.378Z|00020|main|INFO|OVNSB commit failed, force > recompute next time. > 2020-08-05T14:10:05.378Z|00021|jsonrpc|WARN|tcp:253.255.0.33:6642: > receive error: Connection reset by peer > 2020-08-05T14:10:05.378Z|00022|reconnect|WARN|tcp:253.255.0.33:6642: > connection dropped (Connection reset by peer) > 2020-08-05T14:10:05.379Z|00023|reconnect|INFO|tcp:253.255.0.35:6642: > connecting... > 2020-08-05T14:10:05.379Z|00024|reconnect|INFO|tcp:253.255.0.35:6642: > connected > 2020-08-05T14:10:05.485Z|00025|main|INFO|OVNSB commit failed, force > recompute next time. > 2020-08-05T14:10:05.485Z|00026|jsonrpc|WARN|tcp:253.255.0.35:6642: > receive error: Connection reset by peer > 2020-08-05T14:10:05.485Z|00027|reconnect|WARN|tcp:253.255.0.35:6642: > connection dropped (Connection reset by peer) > 2020-08-05T14:10:06.486Z|00028|reconnect|INFO|tcp:253.255.0.34:6642: > connecting... > 2020-08-05T14:10:06.487Z|00029|reconnect|INFO|tcp:253.255.0.34:6642: > connection attempt failed (Connection refused) > 2020-08-05T14:10:06.487Z|00030|reconnect|INFO|tcp:253.255.0.34:6642: > waiting 2 seconds before reconnect > 2020-08-05T14:10:08.488Z|00031|reconnect|INFO|tcp:253.255.0.33:6642: > connecting... > 2020-08-05T14:10:08.488Z|00032|reconnect|INFO|tcp:253.255.0.33:6642: > connection attempt failed (Connection refused) > 2020-08-05T14:10:08.488Z|00033|reconnect|INFO|tcp:253.255.0.33:6642: > waiting 4 seconds before reconnect > 2020-08-05T14:10:12.492Z|00034|reconnect|INFO|tcp:253.255.0.35:6642: > connecting... > 2020-08-05T14:10:12.493Z|00035|reconnect|INFO|tcp:253.255.0.35:6642: > connection attempt failed (Connection refused) > 2020-08-05T14:10:12.493Z|00036|reconnect|INFO|tcp:253.255.0.35:6642: > continuing to reconnect in the background but suppressing further logging > > > > > > On 05/08/2020 20:54, Dumitru Ceara wrote: >> On 8/5/20 5:14 PM, Brendan Doyle wrote: >>> Folks, >>> >>> I'm stumped here, I have the k8 ovnkube-db-raft Stateful set up and >>> running. >>> But when I create a simple network, no SB flows are generated. >>> >>> ovn-nbctl show shows my network. ovn-sbctl show shows the physicals >>> systems in my network. >>> But I can't ping between any hosts because ovn-sbctl lflow-list is >>> empty, and there are no >>> errors or warnings in the logs. The ovn cluster says it is up and >>> healthy. >>> >>> Anybody got any ideas why this might be? >>> >> Hi Brendan, >> >> Maybe I missed it but is ovn-northd running anywhere? If so, could you >> please share its logs? >> >> Thanks, >> Dumitru >> > _______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
