OVN is currently lacking documentation for how to develop tests. This commit adds a file "Documentation/topics/test-development.rst" that provides both a high-level overview of the OVN test-suite, along with documentation for the most commonly used macros and functions.
Reported-at: https://issues.redhat.com/browse/FDP-1548 Signed-off-by: Nicholas Hubbard <[email protected]> --- v2: Implemented changes suggested by Mark and Xavier --- Documentation/automake.mk | 1 + Documentation/topics/index.rst | 1 + Documentation/topics/test-development.rst | 359 ++++++++++++++++++++++ 3 files changed, 361 insertions(+) create mode 100644 Documentation/topics/test-development.rst diff --git a/Documentation/automake.mk b/Documentation/automake.mk index a70c294b9..6bc88c24b 100644 --- a/Documentation/automake.mk +++ b/Documentation/automake.mk @@ -25,6 +25,7 @@ DOC_SOURCE = \ Documentation/tutorials/ovn-interconnection.rst \ Documentation/topics/index.rst \ Documentation/topics/testing.rst \ + Documentation/topics/test-development.rst \ Documentation/topics/high-availability.rst \ Documentation/topics/integration.rst \ Documentation/topics/ovn-news-2.8.rst \ diff --git a/Documentation/topics/index.rst b/Documentation/topics/index.rst index 55bb919c0..896014dc3 100644 --- a/Documentation/topics/index.rst +++ b/Documentation/topics/index.rst @@ -42,6 +42,7 @@ OVN ovn-news-2.8 vif-plug-providers/index testing + test-development .. list-table:: diff --git a/Documentation/topics/test-development.rst b/Documentation/topics/test-development.rst new file mode 100644 index 000000000..823957983 --- /dev/null +++ b/Documentation/topics/test-development.rst @@ -0,0 +1,359 @@ +.. + Licensed under the Apache License, Version 2.0 (the "License"); you may + not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, WITHOUT + WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the + License for the specific language governing permissions and limitations + under the License. + + Convention for heading levels in OVN documentation: + + ======= Heading 0 (reserved for the title in a document) + ------- Heading 1 + ~~~~~~~ Heading 2 + +++++++ Heading 3 + ''''''' Heading 4 + + Avoid deeper levels because they do not render well. + +================ +Test Development +================ + +This document provides information relevant to writing tests for OVN. The +documentation for executing tests exists in :doc:`/topics/testing`. + +OVN uses `Autotest <https://www.gnu.org/software/autoconf/manual/autoconf-2.68/html_node/Using-Autotest.html#Using-Autotest>`_ +for generating its tests. The top-level of the test code is located in the +file ``tests/testsuite.at``. This file is expanded into a shell script that +runs all of the OVN tests. Please refer to the Autotest documentation linked +above for more information regarding Autotest, as the rest of this document +assumes a general understanding of it. + +OVN Test Suite Overview +----------------------- + +All test code is located in the ``tests/`` directory at the root of the OVN +repository. Generally, files suffixed with ``-macros.at`` contain macros and +shell functions to aid in writing tests. Files suffixed with just ``.at`` +contain the code for running actual tests. + +By convention macros are denoted with all uppercase letters, while functions +use lowercase letters. + +The most used and important macros/functions are documented here. All of +these macros/functions are implicitly available from all test files when +running the testsuite via ``make check``. + +Macros and Functions +-------------------- + +Verification +~~~~~~~~~~~~ + +check COMMAND... +++++++++++++++++ + +Function to run COMMAND and check that it succeeds without any output. Also +logs the COMMAND. Note that most ``ovn-nbctl`` and ``ovn-sbctl`` must be run +withing ``check`` so that the return status is checked. + +OVN_CHECK_PACKETS([PCAP], [EXPECTED]) ++++++++++++++++++++++++++++++++++++++ + +Macro to compare packets read from PCAP, in pcap format, to those read from +EXPECTED, which is a text file containing packets as hex strings, one per line. +If PCAP contains fewer packets than EXPECTED, it waits up to 30 seconds by +default for more packets to appear. The wait time can be customized via the +``OVS_CTL_TIMEOUT`` environment variable. + +The reception of any extra or duplicate packets causes the test to fail. + +OVN_CHECK_PACKETS_CONTAIN([PCAP], [EXPECTED]) ++++++++++++++++++++++++++++++++++++++++++++++ + +Macro to check packets read from PCAP contain data from EXPECTED. Waits until +all expected packets are received, and ignores extra packets. + +OVN_CHECK_PACKETS_UNIQ([PCAP], [EXPECTED]) +++++++++++++++++++++++++++++++++++++++++++ + +Macro to check that all EXPECTED packets are received. Duplicate packets are +ignored, however extra packets cause the test to fail. + +OVN_CHECK_PACKETS_REMOVE_BROADCAST([PCAP], [EXPECTED]) +++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +Macro that waits until the expected number of packets (excluding broadcasts) +are received. Then it compares EXPECTED and received packets. Any (non +broadcast), extra or duplicate packets cause the test to fail. + +check_uuid COMMAND +++++++++++++++++++ + +Function to run COMMAND and check that it does not print anything else than +uuid as output. It also fails if the output is empty. + +Daemon/Sandbox Management +~~~~~~~~~~~~~~~~~~~~~~~~~ + +ovn_start [--backup-northd=none|paused] [AZ] +++++++++++++++++++++++++++++++++++++++++++++ + +Creates and initializes ovn-sb and ovn-nb databases and starts their +ovsdb-server instance, sets appropriate environment variables so that ovn-sbctl +and ovn-nbctl use them by default, and starts ovn-northd running against them. + +Normally this starts only an active northd and no backup northd. The following +options are accepted to adjust that: + +``--backup-northd`` Start a backup northd. +``--backup-northd=paused`` Start the backup northd in the paused state. +``--use-tcp-to-sb`` Use TCP instead of SSL to connect to sb database. + +The AZ parameter may name an arbitrary availability zone. If AZ is specified +then the ovn-ic daemon is also started. + +net_add NETWORK ++++++++++++++++ + +Function to create a new interconnection network named NETWORK. + +ovn_attach NETWORK BRIDGE IP [MASKLEN] [ENCAP] +++++++++++++++++++++++++++++++++++++++++++++++ + +First, this function attaches BRIDGE to interconnection network NETWORK. +Second, it configures (simulated) address IP (with network mask length MASKLEN, +which defaults to 24) on BRIDGE. Finally, it configures the Open vSwitch +database to work with OVN and starts ovn-controller. + +sim_add SANDBOX ++++++++++++++++ + +Function to start a new simulated Open vSwitch instance named SANDBOX. Files +related to the instance, such as logs, databases, sockets, and pidfiles, are +created in a subdirectory of the main test directory also named +SANDBOX. Afterward, the ``as`` command (see below) can be used to run Open +vSwitch commands in the context of the new sandbox. + +The new sandbox starts out without any bridges. Use ovs-vsctl in the context of +the new sandbox to create a bridge, e.g.:: + + sim_add hv0 # Create sandbox hv0. + as hv0 # Set hv0 as default sandbox. + ovs-vsctl add-br br0 # Add bridge br0 inside hv0. + +or:: + + sim_add hv0 + as hv0 ovs-vsctl add-br br0 + +as [OVS_DIR] COMMAND +++++++++++++++++++++ + +``as $1`` sets the ``OVS_*DIR`` and ``OVN_*DIR*`` environment variables to +point to $ovs_base/$1. + +``as $1 COMMAND...`` sets those variables in a subshell and invokes COMMAND +there. + +wait_for_ports_up [PORT...] ++++++++++++++++++++++++++++ + +With arguments, this function waits for specified Logical_Switch_Ports to come +up. Without arguments, waits for all "plain" and router Logical_Switch_Ports to +come up. + +PARSE_LISTENING_PORT(LOGFILE, VARIABLE) ++++++++++++++++++++++++++++++++++++++++ + +Macro that parses the TCP or SSL/TLS port on which a server is listening from +LOGFILE, given that the server was told to listen on a kernel-chosen port, and +assigns the port number to shell VARIABLE. You should specify the listening +remote as ptcp:0:127.0.0.1 or pssl:0:127.0.0.1, or the equivalent with [::1] +instead of 127.0.0.1. Here's an example of how to use this with ovsdb-server:: + + ovsdb-server --log-file --remote=ptcp:0:127.0.0.1 ... + PARSE_LISTENING_PORT([ovsdb-server.log], [TCP_PORT]) + +Now $TCP_PORT holds the listening port. + +OVN_POPULATE_ARP() +++++++++++++++++++ + +Macro to pre-populate the ARP tables of all of the OVN instances that have been +started with ```ovn_attach()``. That means that packets sent from one +hypervisor to another never get dropped or delayed by ARP resolution, which +makes testing easier. + +OVS_TRAFFIC_VSWITCHD_START([vsctl-args], [vsctl-output], [=override]) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +Macro to create a database and starts ovsdb-server, starts ovs-vswitchd +connected to that database, calls ovs-vsctl to create a bridge named br0 with +predictable settings, passing 'vsctl-args' as additional commands to +ovs-vsctl. If 'vsctl-args' causes ovs-vsctl to provide output (e.g. because it +includes "create" commands) then 'vsctl-output' specifies the expected output +after filtering through uuidfilt. + +If a test needs to use "system" devices (as dummies), then specify +``=override`` (literally) as the third argument. Otherwise, system devices +won't work at all (which makes sense because tests should not access a system's +real Ethernet devices). + +OVS_TRAFFIC_VSWITCHD_STOP([WHITELIST], [extra_cmds]) +++++++++++++++++++++++++++++++++++++++++++++++++++++ + +Macro to gracefully stops ovs-vswitchd and ovsdb-server, checking their log +files for messages with severity WARN or higher and signaling an error if any +is present. The optional WHITELIST may contain shell-quoted "sed" commands to +delete any warnings that are actually expected, e.g.:: + + OVS_TRAFFIC_VSWITCHD_STOP(["/expected error/d"]) + +'extra_cmds' are shell commands to be executed afte OVS_VSWITCHD_STOP() is +invoked. They can be used to perform additional cleanups such as name space +removal. + +OVS_APP_EXIT_AND_WAIT(DAEMON) ++++++++++++++++++++++++++++++ + +Ask the daemon named DAEMON to exit, via ``ovs-appctl``, and then wait for it +to exit. + +OVN_CLEANUP(sim [, sim ...]) +++++++++++++++++++++++++++++ + +Macro to gracefully terminate all OVN daemons, including those in specified +sandbox instances. Also checks the log file for messages with severity WARN or +higher and signals an error if any is present. Optional arguments may contain +"acceptable" error messages. + +Before terminating the daemons, it also issues recomputes on ovn-controllers in +listed sandboxes, and checks whether the related ports and the openflows before +and after recompute are the same. Optional arguments may also contain +acceptable ``related_ports`` differences, datapaths and tables on which flow +differences are considered as acceptable. + +OVN_CLEANUP_SBOX(sbox) +++++++++++++++++++++++ + +Macro to gracefully terminate OVN daemons in the specified sandbox instance. +The sandbox name ``vtep`` is treated as a special case, and is assumed to have +ovn-controller-vtep and ovs-vtep daemons running instead of ovn-controller. + +Also checks the log file for messages with severity WARN or +higher and signals an error if any is present. Optional arguments may contain +"acceptable" error messages. + +Before terminating the daemons, it also issues recomputes on ovn-controllers in +listed sandboxes, and checks whether the related ports and the openflows before +and after recompute are the same. Optional arguments may also contain +acceptable ``related_ports`` differences, datapaths and tables on which flow +differences are considered as acceptable. + +OVN_CLEANUP_CONTROLLER(sbox) +++++++++++++++++++++++++++++ + +Macro to gracefully terminate ovn-controller in the specified sandbox +instance. The sandbox name ``vtep`` is treated as a special case, and is +assumed to have ovn-controller-vtep and ovs-vtep daemons running instead of +ovn-controller. + +Issues recomputes on ovn-controllers in listed sandbox, and checks whether the +related ports and the openflows before and after recompute are the same. +Optional arguments may also contain acceptable ``related_ports`` differences, +datapaths and tables on which flow differences are considered as acceptable. + +OVN_CLEANUP_IC([az ...]) +++++++++++++++++++++++++ + +Macro to gracefully terminate all interconnection DBs and daemons in the +specified AZs, if any. + +Test Management +~~~~~~~~~~~~~~~ + +OVN_FOR_EACH_NORTHD(TEST) ++++++++++++++++++++++++++ + +Macro to wrap an arbitrary TEST. Defines versions of the TEST with all +combinations of northd, parallelization enabled and conditional monitoring +on/off. Normally the first statement in TEST is a call to ``AT_SETUP``. + +OVN_FOR_EACH_NORTHD_NO_HV(TEST) ++++++++++++++++++++++++++++++++ + +Macro to wrap an arbitrary TEST. Defines versions of the TEST with all +combinations of northd and parallelization enabled. To be used when the +ovn-controller configuration is not relevant. Normally the first statement in +TEST is a call to ``AT_SETUP``. + +on_exit COMMAND ++++++++++++++++ + +Function to add the shell COMMAND to a collection that is executed when the +current test completes, as a cleanup action. The most common use is to kill a +daemon started by the test. This is important to prevent tests that start +daemons from hanging at exit. + +Cleanup commands are executed in the reverse order of calls to this function. + +OVN_NBCTL(NBCTL_COMMAND) +++++++++++++++++++++++++ + +Macro to add NBCTL_COMMAND to list of commands to be run by the +``RUN_OVN_NBCTL`` macro. + +RUN_OVN_NBCTL() ++++++++++++++++ + +Macro to execute a list of commands built by the ``OVN_NBCTL`` macro. The list +of commands is executed in a single invocation of ``ovn-nbctl`` + +OVS_VSCTL(VSCTL_COMMAND) +++++++++++++++++++++++++ + +Macro to add VSCTL_COMMAND to list of commands to be run by ``RUN_OVS_VSCTL``. + +RUN_OVS_VSCTL() ++++++++++++++++ + +Macro to execute the list of commands built by the ``OVS_VSCTL`` macro. The +list of commands is executed in a single invocation of ``ovs-vsctl``. + +OVS_WAIT_FOR_OUTPUT(COMMAND, EXIT-STATUS, STDOUT, STDERR) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +Executes shell COMMAND in a loop until it exits with status EXIT-STATUS, prints +STDOUT on stdout, and prints STDERR on stderr. If this doesn't happen within a +reasonable time limit, then the test fails. + +There is an ``OVS_WAIT_FOR_OUTPUT_UNQUOTED`` version of this macro that expands +shell ``$variables``, ``$(command)``, and so on. The plain version does not + +OVS_WAIT_UNTIL(COMMAND[, IF-FAILED]) +++++++++++++++++++++++++++++++++++++ + +Macro that executes shell COMMAND in a loop until it returns zero return code. +If COMMAND does not return zero code within reasonable time limit, then the +test fails. In that case, runs IF-FAILED before exiting. + +OVS_WAIT_WHILE(COMMAND[, IF-FAILED]) +++++++++++++++++++++++++++++++++++++ + +Macro that executes shell COMMAND in a loop until it returns non-zero return +code. If COMMAND does not return non-zero code within reasonable time limit, +then the test fails. In that case, runs IF-FAILED before exiting. + +OVS_WAIT_UNTIL_EQUAL(COMMAND, OUTPUT) ++++++++++++++++++++++++++++++++++++++ + +Macro that executes shell COMMAND in a loop until it returns zero and the +output equals OUTPUT. If COMMAND does not return zero or a desired output +within a reasonable time limit, fails the test. -- 2.50.1 _______________________________________________ dev mailing list [email protected] https://mail.openvswitch.org/mailman/listinfo/ovs-dev
