But neither IP nor system-id was changed. I've double-checked: ovn-controller 20.06.2:
Chassis "04540082-b5b5-4ab5-9901-03ed445c772d" hostname: host.local Encap vxlan ip: "172.24.33.105" options: {csum="true"} Encap stt ip: "172.24.33.105" options: {csum="true"} Port_Binding eni-3E9901E0 Port_Binding eni-35AFCD00 # ovs-vsctl get open . external-ids:system-id "04540082-b5b5-4ab5-9901-03ed445c772d" # systemctl stop ovn-controller Chassis was deleted: # ovn-sbctl list chassis 04540082-b5b5-4ab5-9901-03ed445c772d ovn-sbctl: no row "04540082-b5b5-4ab5-9901-03ed445c772d" in table Chassis # yum update ovn-host -y # systemctl restart ovn-controller Chassis with same system-id and encap IPs was re-added: Chassis "04540082-b5b5-4ab5-9901-03ed445c772d" hostname: host.local Encap vxlan ip: "172.24.33.105" options: {csum="true"} Encap stt ip: "172.24.33.105" options: {csum="true"} But, there are no port_bindings, and in ovn-controller logs again transaction error: 2020-12-03T12:53:54.031Z|00035|binding|INFO|Claiming lport eni-3E9901E0 for this chassis. 2020-12-03T12:53:54.031Z|00036|binding|INFO|eni-3E9901E0: Claiming 0a:00:3e:99:01:e0 192.168.0.4 2020-12-03T12:53:54.031Z|00037|binding|INFO|Claiming lport eni-35AFCD00 for this chassis. 2020-12-03T12:53:54.031Z|00038|binding|INFO|eni-35AFCD00: Claiming 0a:00:35:af:cd:00 192.168.0.5 2020-12-03T12:53:54.041Z|00039|ovsdb_idl|WARN|transaction error: {"details":"RBAC rules for client \"04540082-b5b5-4ab5-9901-03ed445c772d\" role \"ovn-controller\" prohibit modification of table \"Encap\".","error":"permission error"} 2020-12-03T12:53:54.042Z|00040|main|INFO|OVNSB commit failed, force recompute next time. Moreover, if I forcefully delete chassis, port claim successful, but after restart ovn-controller, promlem appears again: # ovn-sbctl destroy chassis 04540082-b5b5-4ab5-9901-03ed445c772d 2020-12-03T12:56:20.119Z|00045|main|INFO|OVNSB commit failed, force recompute next time. 2020-12-03T12:56:23.803Z|00046|binding|INFO|Claiming lport eni-3E9901E0 for this chassis. 2020-12-03T12:56:23.803Z|00047|binding|INFO|eni-3E9901E0: Claiming 0a:00:3e:99:01:e0 192.168.0.4 2020-12-03T12:56:23.803Z|00048|binding|INFO|Claiming lport eni-35AFCD00 for this chassis. 2020-12-03T12:56:23.803Z|00049|binding|INFO|eni-35AFCD00: Claiming 0a:00:35:af:cd:00 192.168.0.5 # systemctl restart ovn-controller 2020-12-03T12:56:38.590Z|00001|vlog|INFO|opened log file /var/log/ovn/ovn-controller.log 2020-12-03T12:56:38.592Z|00002|reconnect|INFO|unix:/run/openvswitch/db.sock: connecting... 2020-12-03T12:56:38.592Z|00003|reconnect|INFO|unix:/run/openvswitch/db.sock: connected 2020-12-03T12:56:38.596Z|00004|main|INFO|OVS IDL reconnected, force recompute. 2020-12-03T12:56:38.600Z|00005|reconnect|INFO|ssl:x.x.x.x:6642: connecting... 2020-12-03T12:56:38.600Z|00006|main|INFO|OVNSB IDL reconnected, force recompute. 2020-12-03T12:56:38.645Z|00007|reconnect|INFO|ssl:x.x.x.x:6642: connected 2020-12-03T12:56:38.650Z|00008|ofctrl|INFO|unix:/run/openvswitch/br-int.mgmt: connecting to switch 2020-12-03T12:56:38.650Z|00009|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: connecting... 2020-12-03T12:56:38.651Z|00010|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: connected 2020-12-03T12:56:38.654Z|00001|pinctrl(ovn_pinctrl0)|INFO|unix:/run/openvswitch/br-int.mgmt: connecting to switch 2020-12-03T12:56:38.654Z|00002|rconn(ovn_pinctrl0)|INFO|unix:/run/openvswitch/br-int.mgmt: connecting... 2020-12-03T12:56:38.654Z|00011|binding|INFO|Claiming lport eni-35AFCD00 for this chassis. 2020-12-03T12:56:38.654Z|00012|binding|INFO|eni-35AFCD00: Claiming 0a:00:35:af:cd:00 192.168.0.5 2020-12-03T12:56:38.654Z|00013|binding|INFO|Claiming lport eni-3E9901E0 for this chassis. 2020-12-03T12:56:38.654Z|00014|binding|INFO|eni-3E9901E0: Claiming 0a:00:3e:99:01:e0 192.168.0.4 2020-12-03T12:56:38.655Z|00015|ovsdb_idl|WARN|transaction error: {"details":"RBAC rules for client \"04540082-b5b5-4ab5-9901-03ed445c772d\" role \"ovn-controller\" prohibit modification of table \"Encap\".","error":"permission error"} 2020-12-03T12:56:38.655Z|00016|main|INFO|OVNSB commit failed, force recompute next time. Maybe, I just don’t understand your idea... Regards, Vladislav Odintsov On 03.12.2020, 15:38, "Dumitru Ceara" <dce...@redhat.com> wrote: Sorry, I removed the list by accident, readding ovs-dev. On 12/3/20 1:23 PM, Odintsov Vladislav wrote: > Hi Dumitru, > > This helped! > > Chassis destroyed, and port successfully claimed: > > # ovn-sbctl destroy chassis 04540082-b5b5-4ab5-9901-03ed445c772d > > 2020-12-03T12:20:41.222Z|6550427|main|INFO|OVNSB commit failed, force recompute next time. > 2020-12-03T12:20:42.922Z|6550428|binding|INFO|Claiming lport eni-3E9901E0 for this chassis. > 2020-12-03T12:20:42.922Z|6550429|binding|INFO|eni-3E9901E0: Claiming 0a:00:3e:99:01:e0 192.168.0.4 > 2020-12-03T12:20:42.922Z|6550430|binding|INFO|Claiming lport eni-35AFCD00 for this chassis. > 2020-12-03T12:20:42.922Z|6550431|binding|INFO|eni-35AFCD00: Claiming 0a:00:35:af:cd:00 192.168.0.5 > > > Chassis "04540082-b5b5-4ab5-9901-03ed445c772d" > hostname: host.local > Encap vxlan > ip: "X.X.X.X" > options: {csum="true"} > Encap stt > ip: "X.X.X.X" > options: {csum="true"} > Port_Binding eni-3E9901E0 > Port_Binding eni-35AFCD00 > > > But I don't understand what could go wrong? Problem appears right after fresh deploy. I guess the system-id of the chassis changed but the IP didn't. This is an issue that should be documented as the CMS should clear the stale chassis entries if RBAC is enabled [0]. Regards, Dumitru [0] https://mail.openvswitch.org/pipermail/ovs-dev/2020-September/374653.html > > > Regards, > > Vladislav Odintsov > Lead System Engineer at Croc Cloud Development Team > > On 03.12.2020, 15:15, "Dumitru Ceara" <dce...@redhat.com> wrote: > > On 12/3/20 11:50 AM, Odintsov Vladislav wrote: > > Hi, > > > > It seems, that I see regression with claim port functionality in my OVN installation between v20.06.2 and latest branch-20.06 (78174ea) on cluster with enabled rbac. > > Hi Vladislav, > > > > > On v20.06.2 ovn-controller successfully claims port: > > > > # ovn-controller --version > > ovn-controller 20.06.2 > > Open vSwitch Library 2.13.0 > > OpenFlow versions 0x6:0x6 > > > > > > 2020-12-02T18:25:28.787Z|00011|binding|INFO|Claiming lport eni-35AFCD00 for this chassis. > > 2020-12-02T18:25:28.787Z|00012|binding|INFO|eni-35AFCD00: Claiming 0a:00:35:af:cd:00 192.168.0.5 > > 2020-12-02T18:25:28.787Z|00013|binding|INFO|Claiming lport eni-3E9901E0 for this chassis. > > 2020-12-02T18:25:28.787Z|00014|binding|INFO|eni-3E9901E0: Claiming 0a:00:3e:99:01:e0 192.168.0.4 > > > > > > Transaction request: > > > > 2020-12-02T18:50:36.128Z|01605|jsonrpc|DBG|ssl:X.X.X.X:6642: send request, method="transact", params=["OVN_Southbound",{"where":[["_uuid","==",["uuid","4e9bd54c-f083-45cd-93d3-a65f4d20d688"]]],"row":{"chassis":["uuid","9d414bfc-da12-487e-80a0-5c1f2a98a05a"]},"op":"update","table":"Port_Binding"}], id=310 > > > > # ovn-sbctl show | grep 04540082-b5b5-4ab5-9901-03ed445c772d -A 9 > > Chassis "04540082-b5b5-4ab5-9901-03ed445c772d" > > hostname: host.local > > Encap vxlan > > ip: "Y.Y.Y.Y" > > options: {csum="true"} > > Encap stt > > ip: "Y.Y.Y.Y" > > options: {csum="true"} > > Port_Binding eni-3E9901E0 > > Port_Binding eni-35AFCD00 > > > > > > Then I run update OVN (doesn’t matter only ovn controller or full ovn installation): > > > > # ovn-controller --version > > ovn-controller 20.06.3 > > Open vSwitch Library 2.13.0 > > OpenFlow versions 0x6:0x6 > > > > ovn-controller is unable to claim lport: > > > > 2020-12-02T18:53:35.309Z|00043|binding|INFO|Claiming lport eni-3E9901E0 for this chassis. > > 2020-12-02T18:53:35.309Z|00044|binding|INFO|eni-3E9901E0: Claiming 0a:00:3e:99:01:e0 192.168.0.4 > > 2020-12-02T18:53:35.309Z|00045|binding|INFO|Claiming lport eni-DB28C420 for this chassis. > > 2020-12-02T18:53:35.309Z|00046|binding|INFO|eni-DB28C420: Claiming 0a:00:db:28:c4:20 192.168.0.6 > > 2020-12-02T18:53:35.309Z|00047|binding|INFO|Claiming lport eni-35AFCD00 for this chassis. > > 2020-12-02T18:53:35.309Z|00048|binding|INFO|eni-35AFCD00: Claiming 0a:00:35:af:cd:00 192.168.0.5 > > 2020-12-02T18:53:35.345Z|00049|ovsdb_idl|WARN|transaction error: {"details":"RBAC rules for client \"04540082-b5b5-4ab5-9901-03ed445c772d\" role \"ovn-controller\" prohibit modification of table \"Encap\".","error":"permission error"} > > 2020-12-02T18:53:35.345Z|00050|main|INFO|OVNSB commit failed, force recompute next time. > > > > > > Transaction request (added encap table modification comparing to previous version): > > > > 2020-12-02T18:57:33.661Z|20500|jsonrpc|DBG|ssl:X.X.X.X:6642: send request, method="transact", params=["OVN_Southbound",{"where":[["_uuid","==",["uuid","9a397740-4072-4853-9b75-9cc120fe4b34"]]],"row":{"chassis":["uuid","e1436af9-4a15-4480-937e-7584e64033a3"]},"op":"update","table":"Port_Binding"},{"where":[["_uuid","==",["uuid","7ec10f55-c89a-4fd3-a2ab-8ac22f845c85"]]],"row":{"chassis_name":"04540082-b5b5-4ab5-9901-03ed445c772d"},"op":"update","table":"Encap"},{"where":[["_uuid","==",["uuid","256d47ca-ef69-4d75-b967-7ab19bd413a7"]]],"row":{"chassis_name":"04540082-b5b5-4ab5-9901-03ed445c772d"},"op":"update","table":"Encap"},{"where":[["_uuid","==",["uuid","34856b67-7f15-44d3-8071-e20ae0f6029f"]]],"row":{"chassis":["uuid","e1436af9-4a15-4480-937e-7584e64033a3"]},"op":"update","table":"Port_Binding"}], id=113 > > > > > > I’ve configured rbac consulting with this instruction: https://docs.ovn.org/en/latest/tutorials/ovn-rbac.html > > > > Some rbac-related parameters: > > > > # ovn-sbctl list connection > > _uuid : 4940feb2-c4ae-47d9-ade7-6f25c26a2a71 > > external_ids : {} > > inactivity_probe : [] > > is_connected : false > > max_backoff : [] > > other_config : {} > > read_only : false > > role : "" > > status : {} > > target : "pssl:16642" > > > > _uuid : ed9366ef-e352-4210-998f-655f648d638d > > external_ids : {} > > inactivity_probe : [] > > is_connected : false > > max_backoff : [] > > other_config : {} > > read_only : false > > role : ovn-controller > > status : {} > > target : "pssl:6642" > > # ovn-sbctl list rbac_role > > _uuid : 91e9fee1-4aff-4d94-93bf-d4c5119a0dd2 > > name : ovn-controller > > permissions : {Chassis=4a0070bf-1327-4c4d-a7be-83cf91fa1e42, Encap=91da95b4-4eaf-4659-b803-789c72ea3fad, MAC_Binding=660466ef-f0f0-4e58-8be1-a6d16a640ef9, Port_Binding=046836f0-caf1-4d22-88b3-a1d9562d2b58, Service_Monitor=dabca251-6c8e-4953-8769-88f687285a60} > > # ovn-sbctl list rbac_permission > > _uuid : 91da95b4-4eaf-4659-b803-789c72ea3fad > > authorization : [chassis_name] > > insert_delete : true > > table : Encap > > update : [ip, options, type] > > > > _uuid : 046836f0-caf1-4d22-88b3-a1d9562d2b58 > > authorization : [""] > > insert_delete : false > > table : Port_Binding > > update : [chassis] > > > > _uuid : dabca251-6c8e-4953-8769-88f687285a60 > > authorization : [""] > > insert_delete : false > > table : Service_Monitor > > update : [status] > > > > _uuid : 660466ef-f0f0-4e58-8be1-a6d16a640ef9 > > authorization : [""] > > insert_delete : true > > table : MAC_Binding > > update : [datapath, ip, logical_port, mac] > > > > _uuid : 4a0070bf-1327-4c4d-a7be-83cf91fa1e42 > > authorization : [name] > > insert_delete : true > > table : Chassis > > update : [encaps, external_ids, nb_cfg, other_config, vtep_logical_switches] > > > > > > # ovs-vsctl get open . external-ids:system-id > > "04540082-b5b5-4ab5-9901-03ed445c772d" > > # ovs-vsctl get-ssl > > Private key: /var/lib/openvswitch/pki/host/04540082-b5b5-4ab5-9901-03ed445c772d-privkey.pem > > Certificate: /var/lib/openvswitch/pki/host/04540082-b5b5-4ab5-9901-03ed445c772d-cert.pem > > CA Certificate: /var/lib/openvswitch/pki/host/cacert.pem > > Bootstrap: false > > # openssl x509 -noout -subject -in /var/lib/openvswitch/pki/host/04540082-b5b5-4ab5-9901-03ed445c772d-cert.pem > > subject= /C=US/ST=CA/O=Open vSwitch/OU=Open vSwitch certifier/CN=04540082-b5b5-4ab5-9901-03ed445c772d > > > > # ovn-sbctl list chassis 04540082-b5b5-4ab5-9901-03ed445c772d > > _uuid : e1436af9-4a15-4480-937e-7584e64033a3 > > encaps : [256d47ca-ef69-4d75-b967-7ab19bd413a7, 7ec10f55-c89a-4fd3-a2ab-8ac22f845c85] > > external_ids : {datapath-type="", iface-types="erspan,geneve,gre,internal,ip6erspan,ip6gre,lisp,patch,stt,system,tap,vxlan", is-interconn="false", ovn-bridge-mappings="", ovn-chassis-mac-mappings="", ovn-cms-options="", ovn-monitor-all="false"} > > hostname : host.local > > name : "04540082-b5b5-4ab5-9901-03ed445c772d" > > nb_cfg : 0 > > other_config : {datapath-type="", iface-types="erspan,geneve,gre,internal,ip6erspan,ip6gre,lisp,patch,stt,system,tap,vxlan", is-interconn="false", ovn-bridge-mappings="", ovn-chassis-mac-mappings="", ovn-cms-options="", ovn-monitor-all="false"} > > transport_zones : [] > > vtep_logical_switches: [] > > > > # ovn-sbctl list encap 256d47ca-ef69-4d75-b967-7ab19bd413a7 > > _uuid : 256d47ca-ef69-4d75-b967-7ab19bd413a7 > > chassis_name : "04540082-b5b5-4ab5-9901-03ed445c772d" > > ip : "Y.Y.Y.Y" > > options : {csum="true"} > > type : stt > > # ovn-sbctl list encap 7ec10f55-c89a-4fd3-a2ab-8ac22f845c85 > > _uuid : 7ec10f55-c89a-4fd3-a2ab-8ac22f845c85 > > chassis_name : "04540082-b5b5-4ab5-9901-03ed445c772d" > > ip : "Y.Y.Y.Y" > > options : {csum="true"} > > type : vxlan > > > > Can anybody point me what could go wrong? > > Am I missing something? > > > > Could you please try to force ovn-controller to recreate the > Chassis/Chassis_private records after the update? Something like the > following for a chassis with name=04540082-b5b5-4ab5-9901-03ed445c772d: > > ovn-sbctl destroy chassis 04540082-b5b5-4ab5-9901-03ed445c772d > ovn-sbctl destroy chassis_private 04540082-b5b5-4ab5-9901-03ed445c772d > > Thanks, > Dumitru > > > > > Regards, > > > > Vladislav Odintsov > > > > _______________________________________________ > > dev mailing list > > d...@openvswitch.org > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > > > _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev