ovn-controller currently recomputes everything when there are any changes of input, which leads to high CPU usages and slow in end-to-end flow enforcement in response to changes. It even wastes CPU to recompute flows for unrelated inputs such as pinctrl events.
This patch series implements incremental processing in ovn-controller to solve above problems. There has been a similar attempt of solve the problem earlier but was reverted (see commit: 926c34fd). This patch series takes a different approach with an incremental processing engine, to make the dependencies clear and easier to maintain. The engine is a DAG representing dependencies between different nodes. Each node maintains its own data, which depends on its inputs and the data can also be inputs of other nodes. Each node implements a method to recompute its data based on all the inputs, but also implements methods to handle changes of different inputs incrementally. The engine will be responsible to try incremental processing for each node based on the dependencies or fallback to recompute when changes cannot be handled incrementally. This patch series can incrementally process the most common changes: logical flows and port bindings from OVNSB. It can be expanded further for more fine grained incremental processing gradually. In ovn-scale-test env [1], the total execution time of creating and binding 10k ports on 1k HVs with 40 lswitches and 8 lrouters (5 lswitches/lrouter), decreased from 3h40m to 1h50m because of the less CPU on HVs. The CPU time of ovn-controller for additional 500 lports creating and binding (on top of already existed 10k lports) decreased 90% comparing with master. Latency for end-to-end operations of one extra port on top of the 10k lports, start from port-creation until all flows installation on all related HVs is also improved significantly from 20.6s to 7.3s. This is RFC version to get feedback to see if there is any major issue of this approach, before refining it further for formal review. [1] https://github.com/openvswitch/ovn-scale-test --- v1 -> v2: - Fixed the last 2 failed test cases by updates in port-binding change handler - Avoid forced recompute when not needed - Added incremental processing for multicast group changes, which is common during lport creation/deletion - Some other minor fixes and refactors, and rebased on lastest master v2 -> v3: - Better data encapsulation. Avoided using ctx to hold data. - Handling duplicated lflows properly in incremental processing - Fixes for handling meters/groups and conj_id_ofs, etc. - Added documentation for incremental processing engine. - Other refactors and rebased on master Han Zhou (10): ovn-controller: Incremental processing engine ovn-controller: Track OVSDB changes ovn-controller: Initial use of incremental engine in main ovn-controller: Split SB inputs as separate incremental engine nodes ovn-controller: split ovs_idl inputs in incremental engine ovn-controller: Incremental logical flow processing ovn-controller: runtime_data change handler for SB port-binding ovn-controller: port-binding incremental processing for physical flows ovn-controller: Avoid forced recompute when not needed ovn-controller: incremental processing for multicast group changes include/ovn/actions.h | 3 + ovn/controller/bfd.c | 4 +- ovn/controller/binding.c | 101 +++++- ovn/controller/binding.h | 6 + ovn/controller/encaps.c | 12 +- ovn/controller/lflow.c | 178 ++++++++-- ovn/controller/lflow.h | 20 +- ovn/controller/ofctrl.c | 262 ++++++++++----- ovn/controller/ofctrl.h | 34 +- ovn/controller/ovn-controller.c | 726 ++++++++++++++++++++++++++++++---------- ovn/controller/physical.c | 228 +++++++++---- ovn/controller/physical.h | 22 +- ovn/lib/actions.c | 6 +- ovn/lib/automake.mk | 4 +- ovn/lib/extend-table.c | 60 +++- ovn/lib/extend-table.h | 16 +- ovn/lib/inc-proc-eng.c | 144 ++++++++ ovn/lib/inc-proc-eng.h | 228 +++++++++++++ 18 files changed, 1654 insertions(+), 400 deletions(-) create mode 100644 ovn/lib/inc-proc-eng.c create mode 100644 ovn/lib/inc-proc-eng.h -- 2.1.0 _______________________________________________ dev mailing list d...@openvswitch.org https://mail.openvswitch.org/mailman/listinfo/ovs-dev