On 12/3/20 2:01 PM, Odintsov Vladislav wrote: > But neither IP nor system-id was changed. I've double-checked: > > ovn-controller 20.06.2: > > Chassis "04540082-b5b5-4ab5-9901-03ed445c772d" > hostname: host.local > Encap vxlan > ip: "172.24.33.105" > options: {csum="true"} > Encap stt > ip: "172.24.33.105" > options: {csum="true"} > Port_Binding eni-3E9901E0 > Port_Binding eni-35AFCD00 > > # ovs-vsctl get open . external-ids:system-id > "04540082-b5b5-4ab5-9901-03ed445c772d" > > # systemctl stop ovn-controller > > Chassis was deleted: > > # ovn-sbctl list chassis 04540082-b5b5-4ab5-9901-03ed445c772d > ovn-sbctl: no row "04540082-b5b5-4ab5-9901-03ed445c772d" in table Chassis > > # yum update ovn-host -y > # systemctl restart ovn-controller > > Chassis with same system-id and encap IPs was re-added: > > Chassis "04540082-b5b5-4ab5-9901-03ed445c772d" > hostname: host.local > Encap vxlan > ip: "172.24.33.105" > options: {csum="true"} > Encap stt > ip: "172.24.33.105" > options: {csum="true"} > > But, there are no port_bindings, and in ovn-controller logs again transaction > error: > > 2020-12-03T12:53:54.031Z|00035|binding|INFO|Claiming lport eni-3E9901E0 for > this chassis. > 2020-12-03T12:53:54.031Z|00036|binding|INFO|eni-3E9901E0: Claiming > 0a:00:3e:99:01:e0 192.168.0.4 > 2020-12-03T12:53:54.031Z|00037|binding|INFO|Claiming lport eni-35AFCD00 for > this chassis. > 2020-12-03T12:53:54.031Z|00038|binding|INFO|eni-35AFCD00: Claiming > 0a:00:35:af:cd:00 192.168.0.5 > 2020-12-03T12:53:54.041Z|00039|ovsdb_idl|WARN|transaction error: > {"details":"RBAC rules for client \"04540082-b5b5-4ab5-9901-03ed445c772d\" > role \"ovn-controller\" prohibit modification of table > \"Encap\".","error":"permission error"} > 2020-12-03T12:53:54.042Z|00040|main|INFO|OVNSB commit failed, force recompute > next time. > > > Moreover, if I forcefully delete chassis, port claim successful, but after > restart ovn-controller, promlem appears again: > > # ovn-sbctl destroy chassis 04540082-b5b5-4ab5-9901-03ed445c772d > > 2020-12-03T12:56:20.119Z|00045|main|INFO|OVNSB commit failed, force recompute > next time. > 2020-12-03T12:56:23.803Z|00046|binding|INFO|Claiming lport eni-3E9901E0 for > this chassis. > 2020-12-03T12:56:23.803Z|00047|binding|INFO|eni-3E9901E0: Claiming > 0a:00:3e:99:01:e0 192.168.0.4 > 2020-12-03T12:56:23.803Z|00048|binding|INFO|Claiming lport eni-35AFCD00 for > this chassis. > 2020-12-03T12:56:23.803Z|00049|binding|INFO|eni-35AFCD00: Claiming > 0a:00:35:af:cd:00 192.168.0.5 > > # systemctl restart ovn-controller > > 2020-12-03T12:56:38.590Z|00001|vlog|INFO|opened log file > /var/log/ovn/ovn-controller.log > 2020-12-03T12:56:38.592Z|00002|reconnect|INFO|unix:/run/openvswitch/db.sock: > connecting... > 2020-12-03T12:56:38.592Z|00003|reconnect|INFO|unix:/run/openvswitch/db.sock: > connected > 2020-12-03T12:56:38.596Z|00004|main|INFO|OVS IDL reconnected, force recompute. > 2020-12-03T12:56:38.600Z|00005|reconnect|INFO|ssl:x.x.x.x:6642: connecting... > 2020-12-03T12:56:38.600Z|00006|main|INFO|OVNSB IDL reconnected, force > recompute. > 2020-12-03T12:56:38.645Z|00007|reconnect|INFO|ssl:x.x.x.x:6642: connected > 2020-12-03T12:56:38.650Z|00008|ofctrl|INFO|unix:/run/openvswitch/br-int.mgmt: > connecting to switch > 2020-12-03T12:56:38.650Z|00009|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: > connecting... > 2020-12-03T12:56:38.651Z|00010|rconn|INFO|unix:/run/openvswitch/br-int.mgmt: > connected > 2020-12-03T12:56:38.654Z|00001|pinctrl(ovn_pinctrl0)|INFO|unix:/run/openvswitch/br-int.mgmt: > connecting to switch > 2020-12-03T12:56:38.654Z|00002|rconn(ovn_pinctrl0)|INFO|unix:/run/openvswitch/br-int.mgmt: > connecting... > 2020-12-03T12:56:38.654Z|00011|binding|INFO|Claiming lport eni-35AFCD00 for > this chassis. > 2020-12-03T12:56:38.654Z|00012|binding|INFO|eni-35AFCD00: Claiming > 0a:00:35:af:cd:00 192.168.0.5 > 2020-12-03T12:56:38.654Z|00013|binding|INFO|Claiming lport eni-3E9901E0 for > this chassis. > 2020-12-03T12:56:38.654Z|00014|binding|INFO|eni-3E9901E0: Claiming > 0a:00:3e:99:01:e0 192.168.0.4 > 2020-12-03T12:56:38.655Z|00015|ovsdb_idl|WARN|transaction error: > {"details":"RBAC rules for client \"04540082-b5b5-4ab5-9901-03ed445c772d\" > role \"ovn-controller\" prohibit modification of table > \"Encap\".","error":"permission error"} > 2020-12-03T12:56:38.655Z|00016|main|INFO|OVNSB commit failed, force recompute > next time. > > > Maybe, I just don’t understand your idea...
I see. I'm pretty sure it's related to this commit that tries to reuse Encaps (and that's wrong because it doesn't work with RBAC): https://github.com/ovn-org/ovn/commit/94a32fca2d2b825fece0ef5b1873459bd9857dd3 I'll try to fix it and update this thread. Regards, Dumitru > > Regards, > > Vladislav Odintsov > > On 03.12.2020, 15:38, "Dumitru Ceara" <dce...@redhat.com> wrote: > > Sorry, I removed the list by accident, readding ovs-dev. > > On 12/3/20 1:23 PM, Odintsov Vladislav wrote: > > Hi Dumitru, > > > > This helped! > > > > Chassis destroyed, and port successfully claimed: > > > > # ovn-sbctl destroy chassis 04540082-b5b5-4ab5-9901-03ed445c772d > > > > 2020-12-03T12:20:41.222Z|6550427|main|INFO|OVNSB commit failed, force > recompute next time. > > 2020-12-03T12:20:42.922Z|6550428|binding|INFO|Claiming lport > eni-3E9901E0 for this chassis. > > 2020-12-03T12:20:42.922Z|6550429|binding|INFO|eni-3E9901E0: Claiming > 0a:00:3e:99:01:e0 192.168.0.4 > > 2020-12-03T12:20:42.922Z|6550430|binding|INFO|Claiming lport > eni-35AFCD00 for this chassis. > > 2020-12-03T12:20:42.922Z|6550431|binding|INFO|eni-35AFCD00: Claiming > 0a:00:35:af:cd:00 192.168.0.5 > > > > > > Chassis "04540082-b5b5-4ab5-9901-03ed445c772d" > > hostname: host.local > > Encap vxlan > > ip: "X.X.X.X" > > options: {csum="true"} > > Encap stt > > ip: "X.X.X.X" > > options: {csum="true"} > > Port_Binding eni-3E9901E0 > > Port_Binding eni-35AFCD00 > > > > > > But I don't understand what could go wrong? Problem appears right after > fresh deploy. > > I guess the system-id of the chassis changed but the IP didn't. This is > an issue that should be documented as the CMS should clear the stale > chassis entries if RBAC is enabled [0]. > > Regards, > Dumitru > > [0] > https://mail.openvswitch.org/pipermail/ovs-dev/2020-September/374653.html > > > > > > > Regards, > > > > Vladislav Odintsov > > Lead System Engineer at Croc Cloud Development Team > > > > On 03.12.2020, 15:15, "Dumitru Ceara" <dce...@redhat.com> wrote: > > > > On 12/3/20 11:50 AM, Odintsov Vladislav wrote: > > > Hi, > > > > > > It seems, that I see regression with claim port functionality in > my OVN installation between v20.06.2 and latest branch-20.06 (78174ea) on > cluster with enabled rbac. > > > > Hi Vladislav, > > > > > > > > On v20.06.2 ovn-controller successfully claims port: > > > > > > # ovn-controller --version > > > ovn-controller 20.06.2 > > > Open vSwitch Library 2.13.0 > > > OpenFlow versions 0x6:0x6 > > > > > > > > > 2020-12-02T18:25:28.787Z|00011|binding|INFO|Claiming lport > eni-35AFCD00 for this chassis. > > > 2020-12-02T18:25:28.787Z|00012|binding|INFO|eni-35AFCD00: > Claiming 0a:00:35:af:cd:00 192.168.0.5 > > > 2020-12-02T18:25:28.787Z|00013|binding|INFO|Claiming lport > eni-3E9901E0 for this chassis. > > > 2020-12-02T18:25:28.787Z|00014|binding|INFO|eni-3E9901E0: > Claiming 0a:00:3e:99:01:e0 192.168.0.4 > > > > > > > > > Transaction request: > > > > > > 2020-12-02T18:50:36.128Z|01605|jsonrpc|DBG|ssl:X.X.X.X:6642: send > request, method="transact", > params=["OVN_Southbound",{"where":[["_uuid","==",["uuid","4e9bd54c-f083-45cd-93d3-a65f4d20d688"]]],"row":{"chassis":["uuid","9d414bfc-da12-487e-80a0-5c1f2a98a05a"]},"op":"update","table":"Port_Binding"}], > id=310 > > > > > > # ovn-sbctl show | grep 04540082-b5b5-4ab5-9901-03ed445c772d -A 9 > > > Chassis "04540082-b5b5-4ab5-9901-03ed445c772d" > > > hostname: host.local > > > Encap vxlan > > > ip: "Y.Y.Y.Y" > > > options: {csum="true"} > > > Encap stt > > > ip: "Y.Y.Y.Y" > > > options: {csum="true"} > > > Port_Binding eni-3E9901E0 > > > Port_Binding eni-35AFCD00 > > > > > > > > > Then I run update OVN (doesn’t matter only ovn controller or full > ovn installation): > > > > > > # ovn-controller --version > > > ovn-controller 20.06.3 > > > Open vSwitch Library 2.13.0 > > > OpenFlow versions 0x6:0x6 > > > > > > ovn-controller is unable to claim lport: > > > > > > 2020-12-02T18:53:35.309Z|00043|binding|INFO|Claiming lport > eni-3E9901E0 for this chassis. > > > 2020-12-02T18:53:35.309Z|00044|binding|INFO|eni-3E9901E0: > Claiming 0a:00:3e:99:01:e0 192.168.0.4 > > > 2020-12-02T18:53:35.309Z|00045|binding|INFO|Claiming lport > eni-DB28C420 for this chassis. > > > 2020-12-02T18:53:35.309Z|00046|binding|INFO|eni-DB28C420: > Claiming 0a:00:db:28:c4:20 192.168.0.6 > > > 2020-12-02T18:53:35.309Z|00047|binding|INFO|Claiming lport > eni-35AFCD00 for this chassis. > > > 2020-12-02T18:53:35.309Z|00048|binding|INFO|eni-35AFCD00: > Claiming 0a:00:35:af:cd:00 192.168.0.5 > > > 2020-12-02T18:53:35.345Z|00049|ovsdb_idl|WARN|transaction error: > {"details":"RBAC rules for client \"04540082-b5b5-4ab5-9901-03ed445c772d\" > role \"ovn-controller\" prohibit modification of table > \"Encap\".","error":"permission error"} > > > 2020-12-02T18:53:35.345Z|00050|main|INFO|OVNSB commit failed, > force recompute next time. > > > > > > > > > Transaction request (added encap table modification comparing to > previous version): > > > > > > 2020-12-02T18:57:33.661Z|20500|jsonrpc|DBG|ssl:X.X.X.X:6642: send > request, method="transact", > params=["OVN_Southbound",{"where":[["_uuid","==",["uuid","9a397740-4072-4853-9b75-9cc120fe4b34"]]],"row":{"chassis":["uuid","e1436af9-4a15-4480-937e-7584e64033a3"]},"op":"update","table":"Port_Binding"},{"where":[["_uuid","==",["uuid","7ec10f55-c89a-4fd3-a2ab-8ac22f845c85"]]],"row":{"chassis_name":"04540082-b5b5-4ab5-9901-03ed445c772d"},"op":"update","table":"Encap"},{"where":[["_uuid","==",["uuid","256d47ca-ef69-4d75-b967-7ab19bd413a7"]]],"row":{"chassis_name":"04540082-b5b5-4ab5-9901-03ed445c772d"},"op":"update","table":"Encap"},{"where":[["_uuid","==",["uuid","34856b67-7f15-44d3-8071-e20ae0f6029f"]]],"row":{"chassis":["uuid","e1436af9-4a15-4480-937e-7584e64033a3"]},"op":"update","table":"Port_Binding"}], > id=113 > > > > > > > > > I’ve configured rbac consulting with this instruction: > https://docs.ovn.org/en/latest/tutorials/ovn-rbac.html > > > > > > Some rbac-related parameters: > > > > > > # ovn-sbctl list connection > > > _uuid : 4940feb2-c4ae-47d9-ade7-6f25c26a2a71 > > > external_ids : {} > > > inactivity_probe : [] > > > is_connected : false > > > max_backoff : [] > > > other_config : {} > > > read_only : false > > > role : "" > > > status : {} > > > target : "pssl:16642" > > > > > > _uuid : ed9366ef-e352-4210-998f-655f648d638d > > > external_ids : {} > > > inactivity_probe : [] > > > is_connected : false > > > max_backoff : [] > > > other_config : {} > > > read_only : false > > > role : ovn-controller > > > status : {} > > > target : "pssl:6642" > > > # ovn-sbctl list rbac_role > > > _uuid : 91e9fee1-4aff-4d94-93bf-d4c5119a0dd2 > > > name : ovn-controller > > > permissions : > {Chassis=4a0070bf-1327-4c4d-a7be-83cf91fa1e42, > Encap=91da95b4-4eaf-4659-b803-789c72ea3fad, > MAC_Binding=660466ef-f0f0-4e58-8be1-a6d16a640ef9, > Port_Binding=046836f0-caf1-4d22-88b3-a1d9562d2b58, > Service_Monitor=dabca251-6c8e-4953-8769-88f687285a60} > > > # ovn-sbctl list rbac_permission > > > _uuid : 91da95b4-4eaf-4659-b803-789c72ea3fad > > > authorization : [chassis_name] > > > insert_delete : true > > > table : Encap > > > update : [ip, options, type] > > > > > > _uuid : 046836f0-caf1-4d22-88b3-a1d9562d2b58 > > > authorization : [""] > > > insert_delete : false > > > table : Port_Binding > > > update : [chassis] > > > > > > _uuid : dabca251-6c8e-4953-8769-88f687285a60 > > > authorization : [""] > > > insert_delete : false > > > table : Service_Monitor > > > update : [status] > > > > > > _uuid : 660466ef-f0f0-4e58-8be1-a6d16a640ef9 > > > authorization : [""] > > > insert_delete : true > > > table : MAC_Binding > > > update : [datapath, ip, logical_port, mac] > > > > > > _uuid : 4a0070bf-1327-4c4d-a7be-83cf91fa1e42 > > > authorization : [name] > > > insert_delete : true > > > table : Chassis > > > update : [encaps, external_ids, nb_cfg, > other_config, vtep_logical_switches] > > > > > > > > > # ovs-vsctl get open . external-ids:system-id > > > "04540082-b5b5-4ab5-9901-03ed445c772d" > > > # ovs-vsctl get-ssl > > > Private key: > /var/lib/openvswitch/pki/host/04540082-b5b5-4ab5-9901-03ed445c772d-privkey.pem > > > Certificate: > /var/lib/openvswitch/pki/host/04540082-b5b5-4ab5-9901-03ed445c772d-cert.pem > > > CA Certificate: /var/lib/openvswitch/pki/host/cacert.pem > > > Bootstrap: false > > > # openssl x509 -noout -subject -in > /var/lib/openvswitch/pki/host/04540082-b5b5-4ab5-9901-03ed445c772d-cert.pem > > > subject= /C=US/ST=CA/O=Open vSwitch/OU=Open vSwitch > certifier/CN=04540082-b5b5-4ab5-9901-03ed445c772d > > > > > > # ovn-sbctl list chassis 04540082-b5b5-4ab5-9901-03ed445c772d > > > _uuid : e1436af9-4a15-4480-937e-7584e64033a3 > > > encaps : [256d47ca-ef69-4d75-b967-7ab19bd413a7, > 7ec10f55-c89a-4fd3-a2ab-8ac22f845c85] > > > external_ids : {datapath-type="", > iface-types="erspan,geneve,gre,internal,ip6erspan,ip6gre,lisp,patch,stt,system,tap,vxlan", > is-interconn="false", ovn-bridge-mappings="", ovn-chassis-mac-mappings="", > ovn-cms-options="", ovn-monitor-all="false"} > > > hostname : host.local > > > name : "04540082-b5b5-4ab5-9901-03ed445c772d" > > > nb_cfg : 0 > > > other_config : {datapath-type="", > iface-types="erspan,geneve,gre,internal,ip6erspan,ip6gre,lisp,patch,stt,system,tap,vxlan", > is-interconn="false", ovn-bridge-mappings="", ovn-chassis-mac-mappings="", > ovn-cms-options="", ovn-monitor-all="false"} > > > transport_zones : [] > > > vtep_logical_switches: [] > > > > > > # ovn-sbctl list encap 256d47ca-ef69-4d75-b967-7ab19bd413a7 > > > _uuid : 256d47ca-ef69-4d75-b967-7ab19bd413a7 > > > chassis_name : "04540082-b5b5-4ab5-9901-03ed445c772d" > > > ip : "Y.Y.Y.Y" > > > options : {csum="true"} > > > type : stt > > > # ovn-sbctl list encap 7ec10f55-c89a-4fd3-a2ab-8ac22f845c85 > > > _uuid : 7ec10f55-c89a-4fd3-a2ab-8ac22f845c85 > > > chassis_name : "04540082-b5b5-4ab5-9901-03ed445c772d" > > > ip : "Y.Y.Y.Y" > > > options : {csum="true"} > > > type : vxlan > > > > > > Can anybody point me what could go wrong? > > > Am I missing something? > > > > > > > Could you please try to force ovn-controller to recreate the > > Chassis/Chassis_private records after the update? Something like > the > > following for a chassis with > name=04540082-b5b5-4ab5-9901-03ed445c772d: > > > > ovn-sbctl destroy chassis 04540082-b5b5-4ab5-9901-03ed445c772d > > ovn-sbctl destroy chassis_private > 04540082-b5b5-4ab5-9901-03ed445c772d > > > > Thanks, > > Dumitru > > > > > > > > Regards, > > > > > > Vladislav Odintsov > > > > > > _______________________________________________ > > > dev mailing list > > > d...@openvswitch.org > > > https://mail.openvswitch.org/mailman/listinfo/ovs-dev > > > > > > > > > _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev