No worries. Glad to see your problem solved! On Tue, May 5, 2020 at 7:38 PM <fangt...@ruijie.com.cn> wrote:
> Han, > > I am really sorry for wasting your time,someone of us set a source > IP route rule higher than the ovn-ic route ones,which let the pinging > going to the wrong > > destination.as I deleted the src-ip rules,it works. > > Appreciate your help. > > Tian > > > > > > *发件人:* Han Zhou <hz...@ovn.org> > *发送时间:* 2020年5月5日 0:47 > *收件人:* 方田(云桌面 北京) <fangt...@ruijie.com.cn> > *抄送:* Han Zhou <hz...@ovn.org>; ovs-discuss <ovs-discuss@openvswitch.org> > *主题:* Re: [ovs-discuss] 回复: ovn-ic POC problem > > > > > > On Mon, May 4, 2020 at 4:01 AM <fangt...@ruijie.com.cn> wrote: > > > > > > > > > > > > On Sun, May 3, 2020 at 4:18 AM <fangt...@ruijie.com.cn> wrote: > > > > > > > > > > > > > > > > > > On Wed, Apr 29, 2020 at 9:34 PM <fangt...@ruijie.com.cn> wrote: > > > > > > > > > > > > > > > > Hi all, > > > > > > > > I am a developer of Kube-ovn project. Currently I’m doing an > OVN-interconnect POC, the setup script is attached at the bottom. > > > > > > > > My problem is :as everything seems OK, but I can’t ping the Central > AZ from the East AZ. > > > > > > > > I did diagnose this with the ovn-trace tool ,which prompt me with > the message at the end : > > > > > > > > > > > > > > > > “egress(dp="join", inport="join-ovn-cluster", outport="node-east-1") > > > > > > > > ------------------------------------------------------------------- > > > > > > > > 1. ls_out_pre_acl (ovn-northd.c:4690): ip, priority 100, uuid > 7a78dffb > > > > > > > > reg0[0] = 1; > > > > > > > > next; > > > > > > > > 2. ls_out_pre_stateful (ovn-northd.c:4879): reg0[0] == 1, priority > 100, uuid e724b591 > > > > > > > > ct_next; > > > > > > > > > > > > > > > > ct_next(ct_state=est|trk /* default (use --ct to customize) */) > > > > > > > > --------------------------------------------------------------- > > > > > > > > 9. ls_out_port_sec_l2 (ovn-northd.c:4564): outport == "node-east-1", > priority 50, uuid 7c98b562 > > > > > > > > output; > > > > > > > > /* output to "node-east-1", type "" */ > > > > > > > > “ > > > > > > > > Seems the ping package was sent out from the outport "node-east-1" > ,but no response there. > > > > > > > > > > > > > > > > > > > > > > > > Will someone please to tell me what’s wrong with my POC setup ? > > > > > > > > Thanks! > > > > > > > > Tian > > > > > > > > > > > > > > > > > > > > > > > >>It is hard to tell without the full topology. For example, what is > the path between the source and destination? > > > > > > >>Also, what's the command used to generate the ovn-trace output? What > the above trace shows is that the packet is sent out to "node-east-1", not > from. But you mentioned you were pinging from east to central, so I am > confused. > > > > > > >>In addition, did you check the content of IC-SB, with ovn-ic-sbctl > show? Were the ports and GWs populated in IC-SB? > > > > > > > > > > > > >>Thanks, > > > > > > >>Han > > > > > > > > > > > > Thanks for reply ! > > > > > > > > > > > > I did trace the ping data path from the east-1 node,the message is > from the trace command。 > > > > > > My configuration is quite simple,2 AZes with 1 node each. > > > > > > > > > > > > On the central node broker-1 > > > > > > > > > > > > # ovn-ic-nbctl show > > > > > > Transit_Switch ts-aliyun > > > > > > > > > > > > # ovn-ic-sbctl show > > > > > > availability-zone central > > > > > > gateway 785b3dec-0af0-4906-8018-50f511bf4e4f > > > > > > hostname: broker-1 > > > > > > type: geneve > > > > > > ip: 172.17.88.1 > > > > > > port lsp-ovn-cluster-ts-aliyun-central > > > > > > transit switch: ts-aliyun > > > > > > address: ["1a:aa:aa:aa:aa:01 192.168.100.1/24"] > > > > > > availability-zone east > > > > > > gateway d68ac809-292e-4df0-9a45-0254750f2376 > > > > > > hostname: east-1 > > > > > > type: geneve > > > > > > ip: 172.17.88.0 > > > > > > port lsp-ovn-cluster-ts-aliyun-east > > > > > > transit switch: ts-aliyun > > > > > > address: ["1a:aa:aa:aa:aa:02 192.168.100.2/24"] > > > > > > # ovn-sbctl show > > > > > > Chassis "d68ac809-292e-4df0-9a45-0254750f2376" > > > > > > hostname: east-1 > > > > > > Encap geneve > > > > > > ip: "172.17.88.0" > > > > > > options: {csum="true"} > > > > > > Port_Binding lsp-ovn-cluster-ts-aliyun-east > > > > > > Chassis "785b3dec-0af0-4906-8018-50f511bf4e4f" > > > > > > hostname: broker-1 > > > > > > Encap geneve > > > > > > ip: "172.17.88.1" > > > > > > options: {csum="true"} > > > > > > Port_Binding default-http-backend-67cf578fc4-66284.ingress-nginx > > > > > > Port_Binding tiller-deploy-67cd845dff-j7jqx.kube-system > > > > > > Port_Binding coredns-autoscaler-65bfc8d47d-gvvgh.kube-system > > > > > > Port_Binding cr-lrp-ovn-cluster-ts-aliyun-central > > > > > > Port_Binding kube-ovn-pinger-4hq29.kube-system > > > > > > Port_Binding node-broker-1 > > > > > > Port_Binding metrics-server-6b55c64f86-l5cq5.kube-system > > > > > > Port_Binding coredns-7c5566588d-kgjj9.kube-system > > > > > > > > > > > > ovn-sbctl show > > > > > > Chassis "d68ac809-292e-4df0-9a45-0254750f2376" > > > > > > hostname: east-1 > > > > > > Encap geneve > > > > > > ip: "172.17.88.0" > > > > > > options: {csum="true"} > > > > > > Port_Binding lsp-ovn-cluster-ts-aliyun-east > > > > > > Chassis "785b3dec-0af0-4906-8018-50f511bf4e4f" > > > > > > hostname: broker-1 > > > > > > Encap geneve > > > > > > ip: "172.17.88.1" > > > > > > options: {csum="true"} > > > > > > Port_Binding default-http-backend-67cf578fc4-66284.ingress-nginx > > > > > > Port_Binding tiller-deploy-67cd845dff-j7jqx.kube-system > > > > > > Port_Binding coredns-autoscaler-65bfc8d47d-gvvgh.kube-system > > > > > > Port_Binding cr-lrp-ovn-cluster-ts-aliyun-central > > > > > > Port_Binding kube-ovn-pinger-4hq29.kube-system > > > > > > Port_Binding node-broker-1 > > > > > > Port_Binding metrics-server-6b55c64f86-l5cq5.kube-system > > > > > > Port_Binding coredns-7c5566588d-kgjj9.kube-system > > > > > > > > > > > > > > > > > > > > > > > > On the east-1 node , things are similar. > > > > > > > > > > > > sh-4.4# ovn-nbctl show > > > > > > switch 1f157fe4-8576-400e-85fb-7f1fad7daf44 (ovn-default) > > > > > > port metrics-server-6b55c64f86-l9mhb.kube-system > > > > > > addresses: ["00:00:00:A0:C4:B0 10.44.0.6"] > > > > > > port coredns-7c5566588d-bn4fh.kube-system > > > > > > addresses: ["00:00:00:B1:40:ED 10.44.0.3"] > > > > > > port coredns-autoscaler-65bfc8d47d-xk9md.kube-system > > > > > > addresses: ["00:00:00:9C:49:B9 10.44.0.4"] > > > > > > port kube-ovn-pinger-nfcjk.kube-system > > > > > > addresses: ["00:00:00:4A:20:7A 10.44.0.5"] > > > > > > port ovn-default-ovn-cluster > > > > > > type: router > > > > > > addresses: ["00:00:00:DE:08:3C"] > > > > > > router-port: ovn-cluster-ovn-default > > > > > > port default-http-backend-67cf578fc4-wn5mj.ingress-nginx > > > > > > addresses: ["00:00:00:C4:22:11 10.44.0.2"] > > > > > > switch 0cbefdd1-8067-4db2-8a2d-3487dc16c3ea (ts-aliyun) > > > > > > port lsp-ovn-cluster-ts-aliyun-central > > > > > > type: remote > > > > > > addresses: ["1a:aa:aa:aa:aa:01 192.168.100.1/24"] > > > > > > port lsp-ovn-cluster-ts-aliyun-east > > > > > > type: router > > > > > > router-port: lrp-ovn-cluster-ts-aliyun-east > > > > > > switch 61a36d09-0b39-46f1-9da1-967ac195905b (join) > > > > > > port join-ovn-cluster > > > > > > type: router > > > > > > addresses: ["00:00:00:17:31:7A"] > > > > > > router-port: ovn-cluster-join > > > > > > port node-east-1 > > > > > > addresses: ["00:00:00:5F:D7:E6 100.64.0.2"] > > > > > > router b7ef9172-ccf2-4489-8802-a03f1c441a51 (ovn-cluster) > > > > > > port ovn-cluster-join > > > > > > mac: "00:00:00:17:31:7A" > > > > > > networks: ["100.64.0.1/16"] > > > > > > port lrp-ovn-cluster-ts-aliyun-east > > > > > > mac: "1a:aa:aa:aa:aa:02" > > > > > > networks: ["192.168.100.2/24"] > > > > > > gateway chassis: [d68ac809-292e-4df0-9a45-0254750f2376] > > > > > > port ovn-cluster-ovn-default > > > > > > mac: "00:00:00:DE:08:3C" > > > > > > networks: ["10.44.0.1/16"] > > > > > > sh-4.4# ovn-sbctl show > > > > > > Chassis "785b3dec-0af0-4906-8018-50f511bf4e4f" > > > > > > hostname: broker-1 > > > > > > Encap geneve > > > > > > ip: "172.17.88.1" > > > > > > > > options: {csum="true"} > > > > > > > > Here it doesn't show the port-binding for port > lsp-ovn-cluster-ts-aliyun-central, which should be learned from IC-SB DB. > However, the port is shown in NB DB, and the chassis binding is also shown > in IC-SB. So it means the chassis information of the port-binding is not > learned from IC-SB to SB of the east AZ. Could you check if there is any > error log in ovn-ic of east AZ? Are the connections to SB and IC-SB both > working well? Could you share the output of "ovn-sbctl list port_binding" > in east AZ as well? > > > > > > > > > > > > ##East ovn-ic log > > > > > > > > 2020-05-04T10:28:58.129Z|00006|reconnect|INFO|tcp:172.17.88.0:6641: > connected > > > > 2020-05-04T10:28:58.129Z|00007|reconnect|INFO|tcp:172.17.88.0:6642: > connected > > > > 2020-05-04T10:28:58.129Z|00008|reconnect|INFO|tcp:172.17.88.1:6645: > connected > > > > 2020-05-04T10:28:58.129Z|00009|reconnect|INFO|tcp:172.17.88.1:6646: > connected > > > > 2020-05-04T10:28:58.129Z|00010|ovn_ic|INFO|ovn-ic lock acquired. This > ovn-ic instance is now active. > > > > 2020-05-04T10:28:58.130Z|00011|ovn_ic|INFO|NB Global not exist. > > > > 2020-05-04T10:28:58.131Z|00012|ovn_ic|INFO|NB Global not exist. > > > > 2020-05-04T10:28:58.133Z|00013|ovn_ic|WARN|Route sync ignores port > lsp-ovn-cluster-ts-aliyun-east on ts ts-aliyun because logical router port > is not found in NB. > > > > > > > > sh-4.4# ovn-sbctl show > > > > Chassis "785b3dec-0af0-4906-8018-50f511bf4e4f" > > > > hostname: broker-1 > > > > Encap geneve > > > > ip: "172.17.88.1" > > > > options: {csum="true"} > > > > Port_Binding lsp-ovn-cluster-ts-aliyun-central > > > > > > > > > ## this time it has port binding,but still not work. > > > > It is a little strange that the port-binding wasn't there last time. It > could be a transient state or it could be a bug. Could you confirm if the > current DB status is stable? i.e. does every ovn-sbctl show output shows > the same port-bindings? > > > > If it is stable now, the root cause of not pinging could be something > else. Please do tcpdump to firstly identify on which part of the path was > the packet dropped. > > If the ping is from node-east-1 to some node in central AZ, did central > side node receive ping request and sent out reply? Could you do tcpdump on > that node to verify? If not, you can check if the packet arrived iinterconn > GW on each side, by doing tcpdump on the genev_sys_6081interface on the > gateway nodes? If ping is received and reply is sent out by the central AZ > node, you could do similar tcpdump on the GWs to identify which node > dropped the packet. Then the debug can continue there. > > > > > > > > Chassis "d68ac809-292e-4df0-9a45-0254750f2376" > > > > hostname: east-1 > > > > Encap geneve > > > > ip: "172.17.88.0" > > > > options: {csum="true"} > > > > Port_Binding coredns-7c5566588d-bn4fh.kube-system > > > > Port_Binding coredns-autoscaler-65bfc8d47d-xk9md.kube-system > > > > Port_Binding node-east-1 > > > > Port_Binding cr-lrp-ovn-cluster-ts-aliyun-east > > > > Port_Binding metrics-server-6b55c64f86-l9mhb.kube-system > > > > Port_Binding default-http-backend-67cf578fc4-wn5mj.ingress-nginx > > > > Port_Binding kube-ovn-pinger-nfcjk.kube-system > > > > > > > > > > > > sh-4.4# ovn-sbctl list port_binding > > > > _uuid : c9193648-752d-475c-955c-27bdb51bf946 > > > > chassis : 2b10deef-ef4e-4d06-a378-4886c7e0f40e > > > > datapath : 325496e7-112f-40f3-b612-233152629d3c > > > > encap : [] > > > > external_ids : {} > > > > gateway_chassis : [] > > > > ha_chassis_group : [] > > > > logical_port : coredns-7c5566588d-bn4fh.kube-system > > > > mac : ["00:00:00:B1:40:ED 10.44.0.3"] > > > > nat_addresses : [] > > > > options : {} > > > > parent_port : [] > > > > tag : [] > > > > tunnel_key : 3 > > > > type : "" > > > > virtual_parent : [] > > > > > > > > _uuid : c5f6c879-d954-48a2-99b1-8f377c948dde > > > > chassis : 2b10deef-ef4e-4d06-a378-4886c7e0f40e > > > > datapath : 325496e7-112f-40f3-b612-233152629d3c > > > > encap : [] > > > > external_ids : {} > > > > gateway_chassis : [] > > > > ha_chassis_group : [] > > > > logical_port : coredns-autoscaler-65bfc8d47d-xk9md.kube-system > > > > mac : ["00:00:00:9C:49:B9 10.44.0.4"] > > > > nat_addresses : [] > > > > options : {} > > > > parent_port : [] > > > > tag : [] > > > > tunnel_key : 4 > > > > type : "" > > > > virtual_parent : [] > > > > > > > > _uuid : 5d1f8b41-829b-41b1-acd3-00d20a3e27f3 > > > > chassis : [] > > > > datapath : 325496e7-112f-40f3-b612-233152629d3c > > > > encap : [] > > > > external_ids : {} > > > > gateway_chassis : [] > > > > ha_chassis_group : [] > > > > logical_port : ovn-default-ovn-cluster > > > > mac : ["00:00:00:DE:08:3C"] > > > > nat_addresses : [] > > > > options : {peer=ovn-cluster-ovn-default} > > > > parent_port : [] > > > > tag : [] > > > > tunnel_key : 1 > > > > type : patch > > > > virtual_parent : [] > > > > > > > > _uuid : 5e861ab9-5665-4a5e-be91-23e2b3fd52bb > > > > chassis : 2b10deef-ef4e-4d06-a378-4886c7e0f40e > > > > datapath : 34bc4c58-707c-41e7-a5a9-ce9047a0f904 > > > > encap : [] > > > > external_ids : {} > > > > gateway_chassis : [] > > > > ha_chassis_group : [] > > > > logical_port : node-east-1 > > > > mac : ["00:00:00:5F:D7:E6 100.64.0.2"] > > > > nat_addresses : [] > > > > options : {} > > > > parent_port : [] > > > > tag : [] > > > > tunnel_key : 2 > > > > type : "" > > > > virtual_parent : [] > > > > > > > > _uuid : 44857ea4-3be1-4020-b237-51a8894e2db9 > > > > chassis : [] > > > > datapath : fab3717b-6a7c-428f-87b8-a82ddc355f7d > > > > encap : [] > > > > external_ids : {} > > > > gateway_chassis : [] > > > > ha_chassis_group : [] > > > > logical_port : ovn-cluster-join > > > > mac : ["00:00:00:17:31:7A 100.64.0.1/16"] > > > > nat_addresses : [] > > > > options : {peer=join-ovn-cluster} > > > > parent_port : [] > > > > tag : [] > > > > tunnel_key : 1 > > > > type : patch > > > > virtual_parent : [] > > > > > > > > _uuid : 3b30c80d-985c-42ff-93f9-e350eb8192c9 > > > > chassis : 2b10deef-ef4e-4d06-a378-4886c7e0f40e > > > > datapath : fab3717b-6a7c-428f-87b8-a82ddc355f7d > > > > encap : [] > > > > external_ids : {} > > > > gateway_chassis : [] > > > > ha_chassis_group : a3957ffe-9bcc-4aac-a0c0-2f3f2082cfc2 > > > > logical_port : cr-lrp-ovn-cluster-ts-aliyun-east > > > > mac : ["1a:aa:aa:aa:aa:02 192.168.100.2/24"] > > > > nat_addresses : [] > > > > options : {distributed-port=lrp-ovn-cluster-ts-aliyun-east} > > > > parent_port : [] > > > > tag : [] > > > > tunnel_key : 4 > > > > type : chassisredirect > > > > virtual_parent : [] > > > > > > > > _uuid : 829efda5-ddd8-4e72-bef7-8189ae855184 > > > > chassis : 406366bc-64d6-4e68-97d6-25d254009440 > > > > datapath : 44ecebd4-0d9a-44e3-bf25-24213373f503 > > > > encap : [] > > > > external_ids : {} > > > > gateway_chassis : [] > > > > ha_chassis_group : [] > > > > logical_port : lsp-ovn-cluster-ts-aliyun-central > > > > mac : ["1a:aa:aa:aa:aa:01 192.168.100.1/24"] > > > > nat_addresses : [] > > > > options : {requested-tnl-key="1"} > > > > parent_port : [] > > > > tag : [] > > > > tunnel_key : 1 > > > > type : remote > > > > virtual_parent : [] > > > > > > > > _uuid : 1799546d-e41a-4e82-9d1e-038dba468011 > > > > chassis : [] > > > > datapath : fab3717b-6a7c-428f-87b8-a82ddc355f7d > > > > encap : [] > > > > external_ids : {} > > > > gateway_chassis : [] > > > > ha_chassis_group : [] > > > > logical_port : ovn-cluster-ovn-default > > > > mac : ["00:00:00:DE:08:3C 10.44.0.1/16"] > > > > nat_addresses : [] > > > > options : {peer=ovn-default-ovn-cluster} > > > > parent_port : [] > > > > tag : [] > > > > tunnel_key : 2 > > > > type : patch > > > > virtual_parent : [] > > > > > > > > _uuid : d3980235-cd54-43fb-996f-1dc5defaaa02 > > > > chassis : 2b10deef-ef4e-4d06-a378-4886c7e0f40e > > > > datapath : 325496e7-112f-40f3-b612-233152629d3c > > > > encap : [] > > > > external_ids : {} > > > > gateway_chassis : [] > > > > ha_chassis_group : [] > > > > logical_port : metrics-server-6b55c64f86-l9mhb.kube-system > > > > mac : ["00:00:00:A0:C4:B0 10.44.0.6"] > > > > nat_addresses : [] > > > > options : {} > > > > parent_port : [] > > > > tag : [] > > > > tunnel_key : 6 > > > > type : "" > > > > virtual_parent : [] > > > > > > > > _uuid : df777935-63ff-44c4-92d6-1874767cf28b > > > > chassis : [] > > > > datapath : 44ecebd4-0d9a-44e3-bf25-24213373f503 > > > > encap : [] > > > > external_ids : {} > > > > gateway_chassis : [] > > > > ha_chassis_group : [] > > > > logical_port : lsp-ovn-cluster-ts-aliyun-east > > > > mac : [router] > > > > nat_addresses : ["1a:aa:aa:aa:aa:02 192.168.100.2 > is_chassis_resident(\"cr-lrp-ovn-cluster-ts-aliyun-east\")"] > > > > options : {peer=lrp-ovn-cluster-ts-aliyun-east} > > > > parent_port : [] > > > > tag : [] > > > > tunnel_key : 2 > > > > type : patch > > > > virtual_parent : [] > > > > > > > > _uuid : 8b1a6c4e-76b3-4993-8074-b2f1a46968d2 > > > > chassis : [] > > > > datapath : fab3717b-6a7c-428f-87b8-a82ddc355f7d > > > > encap : [] > > > > external_ids : {} > > > > gateway_chassis : [] > > > > ha_chassis_group : [] > > > > logical_port : lrp-ovn-cluster-ts-aliyun-east > > > > mac : ["1a:aa:aa:aa:aa:02 192.168.100.2/24"] > > > > nat_addresses : [] > > > > options : {peer=lsp-ovn-cluster-ts-aliyun-east} > > > > parent_port : [] > > > > tag : [] > > > > tunnel_key : 3 > > > > type : patch > > > > virtual_parent : [] > > > > > > > > _uuid : a163cf6f-c139-4b98-abcd-9b1a4c102e00 > > > > chassis : 2b10deef-ef4e-4d06-a378-4886c7e0f40e > > > > datapath : 325496e7-112f-40f3-b612-233152629d3c > > > > encap : [] > > > > external_ids : {} > > > > gateway_chassis : [] > > > > ha_chassis_group : [] > > > > logical_port : default-http-backend-67cf578fc4-wn5mj.ingress-nginx > > > > mac : ["00:00:00:C4:22:11 10.44.0.2"] > > > > nat_addresses : [] > > > > options : {} > > > > parent_port : [] > > > > tag : [] > > > > tunnel_key : 2 > > > > type : "" > > > > virtual_parent : [] > > > > > > > > _uuid : 473a316f-d4f7-426e-b95b-7381e30e7b79 > > > > chassis : 2b10deef-ef4e-4d06-a378-4886c7e0f40e > > > > datapath : 325496e7-112f-40f3-b612-233152629d3c > > > > encap : [] > > > > external_ids : {} > > > > gateway_chassis : [] > > > > ha_chassis_group : [] > > > > logical_port : kube-ovn-pinger-nfcjk.kube-system > > > > mac : ["00:00:00:4A:20:7A 10.44.0.5"] > > > > nat_addresses : [] > > > > options : {} > > > > parent_port : [] > > > > tag : [] > > > > tunnel_key : 5 > > > > type : "" > > > > virtual_parent : [] > > > > > > > > _uuid : dbafda07-5021-4fb3-9542-fa619e9e2224 > > > > chassis : [] > > > > datapath : 34bc4c58-707c-41e7-a5a9-ce9047a0f904 > > > > encap : [] > > > > external_ids : {} > > > > gateway_chassis : [] > > > > ha_chassis_group : [] > > > > logical_port : join-ovn-cluster > > > > mac : ["00:00:00:17:31:7A"] > > > > nat_addresses : [] > > > > options : {peer=ovn-cluster-join} > > > > parent_port : [] > > > > tag : [] > > > > tunnel_key : 1 > > > > type : patch > > > > virtual_parent : [] > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ > > discuss mailing list > > disc...@openvswitch.org > > https://mail.openvswitch.org/mailman/listinfo/ovs-discuss >
_______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss