On Thu, Aug 6, 2020 at 8:54 AM Venugopal Iyer <venugop...@nvidia.com> wrote:
> Hi, Han: > > > > A comment inline: > > > > *From:* ovn-kuberne...@googlegroups.com <ovn-kuberne...@googlegroups.com> *On > Behalf Of *Han Zhou > *Sent:* Wednesday, August 5, 2020 3:36 PM > *To:* Winson Wang <windson.w...@gmail.com> > *Cc:* ovs-discuss@openvswitch.org; ovn-kuberne...@googlegroups.com; > Dumitru Ceara <dce...@redhat.com>; Han Zhou <hz...@ovn.org> > *Subject:* Re: ovn-k8s scale: how to make new ovn-controller process keep > the previous Open Flow in br-int > > > > *External email: Use caution opening links or attachments* > > > > > > > > On Wed, Aug 5, 2020 at 12:58 PM Winson Wang <windson.w...@gmail.com> > wrote: > > Hello OVN Experts, > > > With ovn-k8s, we need to keep the flows always on br-int which needed by > running pods on the k8s node. > > Is there an ongoing project to address this problem? > > If not, I have one proposal not sure if it is doable. > > Please share your thoughts. > The issue: > > In large scale ovn-k8s cluster there are 200K+ Open Flows on br-int on > every K8s node. When we restart ovn-controller for upgrade using > `ovs-appctl -t ovn-controller exit --restart`, the remaining traffic still > works fine since br-int with flows still be Installed. > > > > However, when a new ovn-controller starts it will connect OVS IDL and do > an engine init run, clearing all OpenFlow flows and install flows based on > SB DB. > > With open flows count above 200K+, it took more than 15 seconds to get > all the flows installed br-int bridge again. > > > Proposal solution for the issue: > > When the ovn-controller gets “exit --start”, it will write a > “ovs-cond-seqno” to OVS IDL and store the value to Open vSwitch table in > external-ids column. When new ovn-controller starts, it will check if the > “ovs-cond-seqno” exists in the Open_vSwitch table, and get the seqno from > OVS IDL to decide if it will force a recomputing process? > > > > > > Hi Winson, > > > > Thanks for the proposal. Yes, the connection break during upgrading is a > real issue in a large scale environment. However, the proposal doesn't > work. The "ovs-cond-seqno" is for the OVSDB IDL for the local conf DB, > which is a completely different connection from the ovs-vswitchd open-flow > connection. > > To avoid clearing the open-flow table during ovn-controller startup, we > can find a way to postpone clearing the OVS flows after the recomputing in > ovn-controller is completed, right before ovn-controller replacing with the > new flows. > > *[vi> ] * > > *[vi> ] Seems like we force recompute today if the OVS IDL is reconnected. > Would it be possible to defer * > > *decision to recompute the flows based on the SB’s nb_cfg we have > sync’d with? i.e. If our nb_cfg is * > > *in sync with the SB’s global nb_cfg, we can skip the recompute? At least > if nothing has changed since* > > *the restart, we won’t need to do anything.. We could stash nb_cfg in OVS > (once ovn-controller receives* > > *conformation from OVS that the physical flows for an nb_cfg update are in > place), which should be cleared if * > > *OVS itself is restarted.. (I mean currently, nb_cfg is used to check if > NB, SB and Chassis are in sync, we * > > *could extend this to OVS/physical flows?)* > nb_cfg is already used by ovn-controller to do that, with the help of "barrier" of OpenFlow, but I am not sure if it 100% working as expected. This basic idea should work, but in practice we need to take care of generating the "installed" flow table and "desired" flow table in ovn-controller. I'd start with "postpone clearing OVS flows" which seems a lower hanging fruit, and then see if any further improvement is needed. > > *Have not thought through this though .. so maybe I am missing something…* > > > > *Thanks,* > > > > *-venu* > > This should largely reduce the time of connection broken during upgrading. > Some changes in the ofctrl module's state machine are required, but I am > not 100% sure if this approach is applicable. Need to check more details. > > > > Thanks, > > Han > > Test log: > > Check flow cnt on br-int every second: > > > > packet_count=0 byte_count=0 flow_count=0 > > packet_count=0 byte_count=0 flow_count=0 > > packet_count=0 byte_count=0 flow_count=0 > > packet_count=0 byte_count=0 flow_count=0 > > packet_count=0 byte_count=0 flow_count=0 > > packet_count=0 byte_count=0 flow_count=0 > > packet_count=0 byte_count=0 flow_count=10322 > > packet_count=0 byte_count=0 flow_count=34220 > > packet_count=0 byte_count=0 flow_count=60425 > > packet_count=0 byte_count=0 flow_count=82506 > > packet_count=0 byte_count=0 flow_count=106771 > > packet_count=0 byte_count=0 flow_count=131648 > > packet_count=2 byte_count=120 flow_count=158303 > > packet_count=29 byte_count=1693 flow_count=185999 > > packet_count=188 byte_count=12455 flow_count=212764 > > > > > > -- > > Winson > > -- > You received this message because you are subscribed to the Google Groups > "ovn-kubernetes" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to ovn-kubernetes+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/ovn-kubernetes/CAMu6iS8eC2EtMJbqBccGD0hyvLFBkzkeJ9sXOsT_TVF3Ltm2hA%40mail.gmail.com > <https://groups.google.com/d/msgid/ovn-kubernetes/CAMu6iS8eC2EtMJbqBccGD0hyvLFBkzkeJ9sXOsT_TVF3Ltm2hA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > -- > You received this message because you are subscribed to the Google Groups > "ovn-kubernetes" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to ovn-kubernetes+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/ovn-kubernetes/CADtzDCn5wEGZZ4%3DdovxhQZ2cgWpEyiPhbChk9amodnxNVgeQxQ%40mail.gmail.com > <https://groups.google.com/d/msgid/ovn-kubernetes/CADtzDCn5wEGZZ4%3DdovxhQZ2cgWpEyiPhbChk9amodnxNVgeQxQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . >
_______________________________________________ discuss mailing list disc...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
