The inc-engine/* ovn-appctl commands (show-stats, clear-stats, recompute, compute-log-timeout, list-stopwatches, enable-stopwatch, disable-stopwatch) were documented for ovn-northd and ovn-controller only; ovn-br-controller and ovn-ic register the same commands but had no man page section describing them. Add the missing sections, and describe each binary's engine_run()/recompute_allowed invocation pattern, and the engine_context structure, in the incremental-processing architecture document.
Also correct ovn-northd(8), which listed the show-stats counter name as "abort" instead of the actual "cancel" (lib/inc-proc-eng.c), and fix a line-wrap nit in ovn-controller(8). Finally, adjust ic-graph width to improve visual layout. Assisted-by: Claude Sonnet 5, Claude Code Signed-off-by: Xavier Simonart <[email protected]> --- v2: Remove non-ASCII bytes. --- .../incremental-processing.rst | 113 ++++++++++++++++-- br-controller/ovn-br-controller.8.xml | 80 +++++++++++++ controller/ovn-controller.8.xml | 4 +- ic/ovn-ic.8.xml | 82 +++++++++++++ northd/ovn-northd.8.xml | 2 +- 5 files changed, 265 insertions(+), 16 deletions(-) diff --git a/Documentation/topics/incremental-processing/incremental-processing.rst b/Documentation/topics/incremental-processing/incremental-processing.rst index bbd7a7501..8986ff3cd 100644 --- a/Documentation/topics/incremental-processing/incremental-processing.rst +++ b/Documentation/topics/incremental-processing/incremental-processing.rst @@ -57,8 +57,11 @@ The engine facilitates: Engine Node State ~~~~~~~~~~~~~~~~~ -The state of an engine node (`enum engine_node_state`) describes the validity -of its data after an engine run: +The whole graph is processed by a single top-level `engine_run(bool +recompute_allowed)` call, typically made once per main loop iteration; this +is distinct from the per-node `run` method described above. The state of an +engine node (`enum engine_node_state`) describes the validity of its data +after such an `engine_run()` call: - `EN_STALE`: The data in the node is not up to date with the DB or its inputs. This is the initial state before the engine runs. @@ -70,6 +73,51 @@ of its data after an engine run: - `EN_CANCELED`: Processing for this node was canceled during the last run. This usually triggers a full recomputation in the next iteration. +Engine Invocation per Binary +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The `recompute_allowed` argument to that call is what determines whether +cancellation can happen: a node is canceled when it is marked as writing to +the Southbound DB (`sb_write`) and needs a full recompute while the caller +passed `recompute_allowed = false`. Cancellation stops that engine run +immediately; the whole graph is retried once `recompute_allowed` is `true` +again on some later iteration, which need not be the very next one. + +`ovn-northd` calls `engine_run()` only once it holds the Southbound DB lock +and has valid Northbound and Southbound transactions in hand. `ovn-ic` +similarly checks that all four of its NB/SB/IC-NB/IC-SB transactions are +non-NULL. Otherwise, they skip running the engine for that iteration +entirely. Because of this, they always pass `recompute_allowed = true` and +their nodes are never canceled. + +`ovn-br-controller`'s nodes are never canceled either: it connects to two +databases -- the local `Open_vSwitch` database and the separate OVN Bridge +database -- but only reads from them. Unlike `ovn-northd` and `ovn-ic` +though, this isn't an enforced guarantee: its precondition for calling +`engine_run()` doesn't actually check for a valid transaction, it simply +happens that no node needs one today. + +`ovn-controller` takes a different approach: it calls `engine_run()` on +every main loop iteration regardless of whether a full recompute is +currently possible, so that purely incremental changes can still be +processed even when a full recompute isn't. It passes +`recompute_allowed = true` only when it has a Southbound DB transaction +and no backlog of unsent OpenFlow messages; otherwise it passes +`recompute_allowed = false`, for two reasons: + +- It has no Southbound DB transaction yet this iteration. If, during that + run, a node marked `sb_write` has an input that changed and needs a full + recompute (no change handler for it, or its change handler returned + `EN_UNHANDLED` -- see "Run Function" above), the engine cancels that node + instead of calling its `run` method. A recompute would have been unsafe + anyway: a write attempted with no transaction can't be caught by the + engine's write-protection check. +- `ovs-vswitchd` isn't necessarily unresponsive, but `ovn-controller` still + has OpenFlow messages from a previous round that it hasn't managed to hand + off to the socket connecting to it yet. Rather than wait for any + acknowledgment, it simply defers adding more work on top of what's already + queued. This is a deliberate optimization, not a hard requirement. + Input Handler Result ~~~~~~~~~~~~~~~~~~~~ @@ -97,9 +145,14 @@ DOT format. Dumping the Graph ~~~~~~~~~~~~~~~~~ -Both `ovn-northd`, `ovn-controller` and `ovn-ic` support the +`ovn-northd`, `ovn-controller` and `ovn-ic` all support the `--dump-inc-proc-graph` command-line option. +.. note:: + `ovn-br-controller` also builds its desired state (bridge flows) using + the incremental processing engine, but it does not currently implement + `--dump-inc-proc-graph`. + Full Graph Dump +++++++++++++++ @@ -165,8 +218,10 @@ from Graphviz: ovn-ic --dump-inc-proc-graph | dot -Tpng -o engine_graph.png -Visual representation of the ovn-northd incremental processing engine -dependencies up to the `datapath_sync` node: +As an example, here is each binary's graph rendered up to one representative +node: + +`ovn-northd`, up to the `datapath_sync` node: .. code-block:: bash @@ -177,9 +232,7 @@ dependencies up to the `datapath_sync` node: :width: 80% :align: center - -Visual representation of the ovn-controller incremental processing engine -dependencies up to the `evpn_arp` node: +`ovn-controller`, up to the `evpn_arp` node: .. code-block:: bash @@ -190,9 +243,7 @@ dependencies up to the `evpn_arp` node: :width: 80% :align: center - -Visual representation of the ovn-ic incremental processing engine -dependencies up to the `ic` node: +`ovn-ic`, up to the `ic` node: .. code-block:: bash @@ -200,9 +251,36 @@ dependencies up to the `ic` node: .. figure:: ic-graph.png :alt: OVN Incremental Processing Graph (ic node) - :width: 80% + :width: 25% :align: center +Runtime Introspection +--------------------- + +`ovn-northd`, `ovn-controller`, `ovn-ic` and `ovn-br-controller` all register +the same set of `ovn-appctl`/`ovs-appctl` commands for inspecting and +controlling the engine at runtime: + +- `inc-engine/show-stats [node] [counter]`: Display the per-node + `recompute`, `compute` and `cancel` counters. If `node` is given, only + that node's counters are shown; `counter` further restricts the output to + one of `recompute`, `compute` or `cancel`. +- `inc-engine/clear-stats`: Reset all engine counters to zero. +- `inc-engine/recompute`: Force a full recompute of the engine on the next + iteration, bypassing incremental change handlers. +- `inc-engine/compute-log-timeout <msecs>`: Set the threshold, in + milliseconds, above which a node's compute/recompute time is logged as + slow (default 500ms). +- `inc-engine/list-stopwatches [node]`: List engine nodes and, for each, + the change handlers that can be timed. If `node` is given, only that + node's handlers are listed. +- `inc-engine/enable-stopwatch [node]`: Start timing recompute and change + handler invocations for `node`, or for every node if omitted. Results are + viewed with the `stopwatch/show` command. Stopwatches are disabled by + default, as they add measurable CPU overhead. +- `inc-engine/disable-stopwatch [node]`: Stop timing for `node`, or for + every node if omitted. This is the default state. + Implementation Details ---------------------- @@ -213,9 +291,18 @@ Key Structures ~~~~~~~~~~~~~~ - `struct engine_node`: The base structure for a node. Contains inputs, data, - state, and callback functions (`run`, `init`, `cleanup`). + state, callback functions (`run`, `init`, `cleanup`, `is_valid`, + `clear_tracked_data`, `get_compute_failure_info`), the per-node `stats` + counters exposed by `inc-engine/show-stats`, and the `stopwatch_enabled` + flag toggled by `inc-engine/enable-stopwatch` and + `inc-engine/disable-stopwatch`. - `struct engine_node_input`: Describes a dependency on another node and the associated change handler. +- `struct engine_context`: Carries the OVSDB transaction handles for the + current iteration, plus an opaque `client_ctx` pointer that each binary + uses to pass its own private state (e.g. `ovn-controller`'s + `controller_engine_ctx`) through to `run()` methods and change handlers. + Accessed with `engine_get_context()`. Usage Guidelines ---------------- diff --git a/br-controller/ovn-br-controller.8.xml b/br-controller/ovn-br-controller.8.xml index 6816ece8f..77370d977 100644 --- a/br-controller/ovn-br-controller.8.xml +++ b/br-controller/ovn-br-controller.8.xml @@ -76,4 +76,84 @@ </p> </dd> </dl> + + <h1>Runtime Management Commands</h1> + <p> + <code>ovn-appctl</code> can send commands to a running + <code>ovn-br-controller</code> process. The currently supported + commands are described below. + <dl> + <dt><code>inc-engine/show-stats</code></dt> + <dd> + Display <code>ovn-br-controller</code> engine counters. For each + engine node the following counters have been added: + <ul> + <li> + <code>recompute</code> + </li> + <li> + <code>compute</code> + </li> + <li> + <code>cancel</code> + </li> + </ul> + </dd> + + <dt><code>inc-engine/show-stats</code> + <var>engine_node_name</var> [<var>counter_name</var>]</dt> + <dd> + <p> + Display the <code>ovn-br-controller</code> engine counter(s) for the + specified <var>engine_node_name</var>. <var>counter_name</var> is + optional and can be one of <code>recompute</code>, + <code>compute</code> or <code>cancel</code>. + </p> + </dd> + + <dt><code>inc-engine/clear-stats</code></dt> + <dd> + Reset <code>ovn-br-controller</code> engine counters. + </dd> + + <dt><code>inc-engine/recompute</code></dt> + <dd> + Triggers a full recompute of the incremental processing engine + on the next iteration. + </dd> + + <dt><code>inc-engine/compute-log-timeout</code> <var>msecs</var></dt> + <dd> + Sets the timeout in milliseconds for logging engine compute + events. + </dd> + + <dt><code>inc-engine/list-stopwatches</code> [<var>node</var>]</dt> + <dd> + Lists the engine nodes and their change handler names. If + <var>node</var> is specified, only the handlers for that engine + node are listed. + </dd> + + <dt><code>inc-engine/enable-stopwatch</code> [<var>node</var>]</dt> + <dd> + Enables the per-node and per-handler stopwatches in the + incremental processing engine. If <var>node</var> is specified, + only that node's stopwatches are enabled; otherwise all nodes + are affected. While enabled, every engine node recompute and + change handler invocation is timed and the results can be + viewed with the <code>stopwatch/show</code> command. + Stopwatches are disabled by default because they add measurable + CPU overhead. + </dd> + + <dt><code>inc-engine/disable-stopwatch</code> [<var>node</var>]</dt> + <dd> + Disables the per-node and per-handler stopwatches in the + incremental processing engine. If <var>node</var> is specified, + only that node's stopwatches are disabled; otherwise all nodes + are affected. This is the default state. + </dd> + </dl> + </p> </manpage> diff --git a/controller/ovn-controller.8.xml b/controller/ovn-controller.8.xml index 3c33654ff..2b90f25d9 100644 --- a/controller/ovn-controller.8.xml +++ b/controller/ovn-controller.8.xml @@ -1157,8 +1157,8 @@ <dt><code>inc-engine/list-stopwatches</code> [<var>node</var>]</dt> <dd> Lists the engine nodes and their change handler names. If - <var>node</var> is specified, only the handlers for that engine node - are listed. + <var>node</var> is specified, only the handlers for that engine + node are listed. </dd> <dt><code>inc-engine/enable-stopwatch</code> [<var>node</var>]</dt> diff --git a/ic/ovn-ic.8.xml b/ic/ovn-ic.8.xml index 2153a4a17..7e83d245c 100644 --- a/ic/ovn-ic.8.xml +++ b/ic/ovn-ic.8.xml @@ -44,6 +44,16 @@ its value is used as the default. Otherwise, the default is <code>unix:@RUNDIR@/ovn_ic_sb_db.sock</code>. </dd> + <dt><code>--dump-inc-proc-graph</code> + [<code>=<var>node</var></code>]</dt> + <dd> + <p> + Dump the incremental processing engine graph representation in + DOT format to stdout at startup and then exit. If + <var>node</var> is specified, only the subgraph rooted at that + engine node is printed. + </p> + </dd> </dl> <p> <var>database</var> in the above options must be an OVSDB active or @@ -104,6 +114,78 @@ acquired OVSDB lock on SB DB, "standby" if it has not or "paused" if this instance is paused. </dd> + + <dt><code>inc-engine/show-stats</code></dt> + <dd> + Display <code>ovn-ic</code> engine counters. For each engine + node the following counters have been added: + <ul> + <li> + <code>recompute</code> + </li> + <li> + <code>compute</code> + </li> + <li> + <code>cancel</code> + </li> + </ul> + </dd> + + <dt><code>inc-engine/show-stats</code> + <var>engine_node_name</var> [<var>counter_name</var>]</dt> + <dd> + <p> + Display the <code>ovn-ic</code> engine counter(s) for the + specified <var>engine_node_name</var>. <var>counter_name</var> is + optional and can be one of <code>recompute</code>, + <code>compute</code> or <code>cancel</code>. + </p> + </dd> + + <dt><code>inc-engine/clear-stats</code></dt> + <dd> + Reset <code>ovn-ic</code> engine counters. + </dd> + + <dt><code>inc-engine/recompute</code></dt> + <dd> + Triggers a full recompute of the incremental processing engine + on the next iteration. + </dd> + + <dt><code>inc-engine/compute-log-timeout</code> <var>msecs</var></dt> + <dd> + Sets the timeout in milliseconds for logging engine compute + events. + </dd> + + <dt><code>inc-engine/list-stopwatches</code> [<var>node</var>]</dt> + <dd> + Lists the engine nodes and their change handler names. If + <var>node</var> is specified, only the handlers for that engine + node are listed. + </dd> + + <dt><code>inc-engine/enable-stopwatch</code> [<var>node</var>]</dt> + <dd> + Enables the per-node and per-handler stopwatches in the + incremental processing engine. If <var>node</var> is specified, + only that node's stopwatches are enabled; otherwise all nodes + are affected. While enabled, every engine node recompute and + change handler invocation is timed and the results can be + viewed with the <code>stopwatch/show</code> command. + Stopwatches are disabled by default because they add measurable + CPU overhead. + </dd> + + <dt><code>inc-engine/disable-stopwatch</code> [<var>node</var>]</dt> + <dd> + Disables the per-node and per-handler stopwatches in the + incremental processing engine. If <var>node</var> is specified, + only that node's stopwatches are disabled; otherwise all nodes + are affected. This is the default state. + </dd> </dl> </p> diff --git a/northd/ovn-northd.8.xml b/northd/ovn-northd.8.xml index c2670601a..f4ad26073 100644 --- a/northd/ovn-northd.8.xml +++ b/northd/ovn-northd.8.xml @@ -229,7 +229,7 @@ Display the <code>ovn-northd</code> engine counter(s) for the specified <var>engine_node_name</var>. <var>counter_name</var> is optional and can be one of <code>recompute</code>, <code>compute</code> or - <code>abort</code>. + <code>cancel</code>. </p> </dd> -- 2.56.0 _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
