Hi, Han: A comment inline:
From: ovn-kuberne...@googlegroups.com <ovn-kuberne...@googlegroups.com> On Behalf Of Han Zhou Sent: Wednesday, August 5, 2020 3:36 PM To: Winson Wang <windson.w...@gmail.com> Cc: ovs-discuss@openvswitch.org; ovn-kuberne...@googlegroups.com; Dumitru Ceara <dce...@redhat.com>; Han Zhou <hz...@ovn.org> Subject: Re: ovn-k8s scale: how to make new ovn-controller process keep the previous Open Flow in br-int External email: Use caution opening links or attachments On Wed, Aug 5, 2020 at 12:58 PM Winson Wang <windson.w...@gmail.com<mailto:windson.w...@gmail.com>> wrote: Hello OVN Experts, With ovn-k8s, we need to keep the flows always on br-int which needed by running pods on the k8s node. Is there an ongoing project to address this problem? If not, I have one proposal not sure if it is doable. Please share your thoughts. The issue: In large scale ovn-k8s cluster there are 200K+ Open Flows on br-int on every K8s node. When we restart ovn-controller for upgrade using `ovs-appctl -t ovn-controller exit --restart`, the remaining traffic still works fine since br-int with flows still be Installed. However, when a new ovn-controller starts it will connect OVS IDL and do an engine init run, clearing all OpenFlow flows and install flows based on SB DB. With open flows count above 200K+, it took more than 15 seconds to get all the flows installed br-int bridge again. Proposal solution for the issue: When the ovn-controller gets “exit --start”, it will write a “ovs-cond-seqno” to OVS IDL and store the value to Open vSwitch table in external-ids column. When new ovn-controller starts, it will check if the “ovs-cond-seqno” exists in the Open_vSwitch table, and get the seqno from OVS IDL to decide if it will force a recomputing process? Hi Winson, Thanks for the proposal. Yes, the connection break during upgrading is a real issue in a large scale environment. However, the proposal doesn't work. The "ovs-cond-seqno" is for the OVSDB IDL for the local conf DB, which is a completely different connection from the ovs-vswitchd open-flow connection. To avoid clearing the open-flow table during ovn-controller startup, we can find a way to postpone clearing the OVS flows after the recomputing in ovn-controller is completed, right before ovn-controller replacing with the new flows. [vi> ] [vi> ] Seems like we force recompute today if the OVS IDL is reconnected. Would it be possible to defer decision to recompute the flows based on the SB’s nb_cfg we have sync’d with? i.e. If our nb_cfg is in sync with the SB’s global nb_cfg, we can skip the recompute? At least if nothing has changed since the restart, we won’t need to do anything.. We could stash nb_cfg in OVS (once ovn-controller receives conformation from OVS that the physical flows for an nb_cfg update are in place), which should be cleared if OVS itself is restarted.. (I mean currently, nb_cfg is used to check if NB, SB and Chassis are in sync, we could extend this to OVS/physical flows?) Have not thought through this though .. so maybe I am missing something… Thanks, -venu This should largely reduce the time of connection broken during upgrading. Some changes in the ofctrl module's state machine are required, but I am not 100% sure if this approach is applicable. Need to check more details. Thanks, Han Test log: Check flow cnt on br-int every second: packet_count=0 byte_count=0 flow_count=0 packet_count=0 byte_count=0 flow_count=0 packet_count=0 byte_count=0 flow_count=0 packet_count=0 byte_count=0 flow_count=0 packet_count=0 byte_count=0 flow_count=0 packet_count=0 byte_count=0 flow_count=0 packet_count=0 byte_count=0 flow_count=10322 packet_count=0 byte_count=0 flow_count=34220 packet_count=0 byte_count=0 flow_count=60425 packet_count=0 byte_count=0 flow_count=82506 packet_count=0 byte_count=0 flow_count=106771 packet_count=0 byte_count=0 flow_count=131648 packet_count=2 byte_count=120 flow_count=158303 packet_count=29 byte_count=1693 flow_count=185999 packet_count=188 byte_count=12455 flow_count=212764 -- Winson -- You received this message because you are subscribed to the Google Groups "ovn-kubernetes" group. To unsubscribe from this group and stop receiving emails from it, send an email to ovn-kubernetes+unsubscr...@googlegroups.com<mailto:ovn-kubernetes+unsubscr...@googlegroups.com>. To view this discussion on the web visit https://groups.google.com/d/msgid/ovn-kubernetes/CAMu6iS8eC2EtMJbqBccGD0hyvLFBkzkeJ9sXOsT_TVF3Ltm2hA%40mail.gmail.com<https://groups.google.com/d/msgid/ovn-kubernetes/CAMu6iS8eC2EtMJbqBccGD0hyvLFBkzkeJ9sXOsT_TVF3Ltm2hA%40mail.gmail.com?utm_medium=email&utm_source=footer>. -- You received this message because you are subscribed to the Google Groups "ovn-kubernetes" group. To unsubscribe from this group and stop receiving emails from it, send an email to ovn-kubernetes+unsubscr...@googlegroups.com<mailto:ovn-kubernetes+unsubscr...@googlegroups.com>. To view this discussion on the web visit https://groups.google.com/d/msgid/ovn-kubernetes/CADtzDCn5wEGZZ4%3DdovxhQZ2cgWpEyiPhbChk9amodnxNVgeQxQ%40mail.gmail.com<https://groups.google.com/d/msgid/ovn-kubernetes/CADtzDCn5wEGZZ4%3DdovxhQZ2cgWpEyiPhbChk9amodnxNVgeQxQ%40mail.gmail.com?utm_medium=email&utm_source=footer>.
_______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
