JDevlieghere updated this revision to Diff 257840. JDevlieghere marked 6 inline comments as done. JDevlieghere added a comment.
Thanks for the review! CHANGES SINCE LAST ACTION https://reviews.llvm.org/D78242/new/ https://reviews.llvm.org/D78242 Files: lldb/docs/resources/test.rst
Index: lldb/docs/resources/test.rst =================================================================== --- lldb/docs/resources/test.rst +++ lldb/docs/resources/test.rst @@ -1,27 +1,158 @@ Testing ======= +.. contents:: + :local: + +Test Suite Structure +-------------------- + The LLDB test suite consists of three different kinds of test: -* Unit test. These are located under ``lldb/unittests`` and are written in C++ - using googletest. -* Integration tests that test the debugger through the SB API. These are - located under ``lldb/packages/Python/lldbsuite`` and are written in Python - using ``dotest`` (LLDB's custom testing framework on top of unittest2). -* Integration tests that test the debugger through the command line. These are - located under `lldb/test/Shell` and are written in a shell-style format - using FileCheck to verify its output. +* **Unit tests**: written in C++ using the googletest unit testing library. +* **Shell tests**: Integration tests that test the debugger through the command + line. These tests interact with the debugger either through the command line + driver or through ``lldb-test`` which is a tool that exposes the internal + data structures in an easy-to-parse way for testing. Most people will know + these as *lit tests* in LLVM, although lit is the test driver and ShellTest + is the test format that uses ``RUN:`` lines. ``FileCheck`` is used to verify + the output. +* **API tests**: Integration tests that interact with the debugger through the + SB API. These are are written in Python and use LLDB's ``dotest.py`` testing + framework on top of Python's unittest2. + +All three test suites use ``lit`` (`LLVM Integrated Tester +<https://llvm.org/docs/CommandGuide/lit.html>`_ ) as the test driver. The test +suites can be run as a whole or separately. + + +Unit Tests +`````````` + +Unit tests are located under ``lldb/unittests``. If it's possible to test +something in isolation or as a single unit, you should make it a unit test. + +Often you need instances of the core objects such as a debugger, target or +process, in order to test something meaningful. We already have a handful of +tests that have the necessary boiler plate, but this is something we could +abstract away and make it more user friendly. + +Shell Tests +``````````` + +Shell tests are located under ``lldb/test/shell``. These tests are generally +build around checking the output of ``lldb`` (the command line driver) or +``lldb-test`` using ``FileCheck``. Shell tests are generally small and fast to +write because they require little boilerplate. + +``lldb-test`` is a relatively new addition to the test suite. It was the first +tool that was added that is designed for testing. Since then it has been +continuously extended with new subcommands, improving our test coverage. Among +other things you can use it to query lldb for symbol files, for object files +and breakpoints. + +Obviously shell tests are great for testing the command line driver itself or +the subcomponents already exposed by lldb-test. But when it comes to LLDB's +vast functionality, most things can be tested both through the driver as well +as the Python API. For example, to test setting a breakpoint, you could do it +from the command line driver with ``b main``` or you could use the SB API and do +something like ``target.BreakpointCreateByName``. + +A good rule of thumb is to prefer shell tests when what is being tested is +relatively simple. Expressivity is limited compared to the API tests, which +means that you have to have a well-defined test scenario that you can easily +match with ``FileCheck``. + +Another thing to consider are the binaries being debugged, which we call +inferiors. For shell tests, they have to be relatively simple. The +``dotest.py`` test framework has extensive support for complex build scenarios +and different variants, which is described in more detail below, while shell +tests are limited to single lines of shell commands with compiler and linker +invocations. + +Finally, the shell tests always run in batch mode. You start with some input +and the test verifies the output. The debugger can be sensitive to its +environment, such as the the platform it runs on. It can be hard to express +that the same test might behave slightly differently on macOS and Linux. +Additionally, the debugger is an interactive tool, and the shell test provide +no good way of testing those interactive aspects, such as tab completion for +example. + +API Tests +````````` + +API tests are located under ``lldb/test/API``. Thy are run with the +``dotest.py``. Tests are written in Python and test binaries (inferiors) are +compiled with Make. The majority of API tests are end-to-end tests that compile +programs from source, run them, and debug the processes. + +Below is the directory layout of the example API test. The test directory will +always contain a python file, starting with ``Test``. Most of the tests are +structured as a binary begin debugged, so there will be one or more sources +file and a Makefile. -All three test suites use the `LLVM Integrated Tester -<https://llvm.org/docs/CommandGuide/lit.html>`_ (lit) as their test driver. The -test suites can be run as a whole or separately. +:: -Many of the tests are accompanied by a C (C++, ObjC, etc.) source file. Each -test first compiles the source file and then uses LLDB to debug the resulting -executable. + sample_test + ├── Makefile + ├── TestSampleTest.py + └── main.c + +Let's start with the Python test file. Every test is its own class and can have +one or more test methods, that start with ``test_``. Many tests define +multiple test methods and share a bunch of common code. For example, for a +fictive tests that makes sure we can set breakpoints we might have one test +method that ensures we can set a breakpoint by address, on that sets a +breakpoint by name and another that sets the same breakpoint by file and line +number. The setup, teardown and everything else other than setting the +breakpoint could be shared. + +Our testing framework also has a bunch of utilities that abstract common +operations, such as creating targets, setting breakpoints etc. When code is +shared across tests, we extract it into a utility in ``lldbutil``. It's always +worth taking a look at lldbutil to see if there's a utility to simplify some +of the testing boiler plate. Because we can't always audit every existing test, +this is double true when looking at an existing test for inspiration. + +It's possible to skip or XFAIL tests using decorators. You'll see them a lot. +The debugger can be sensitive to things like the architecture, the host and +target platform, the compiler version etc. LLDB comes with a range of +predefined decorators for these configurations. Another great thing about these +decorators is that they're very easy to extend, it's even possible to define a +function in a test case that determines whether the test should be run or not. + +In addition to providing a lot more flexibility when it comes to writing the +test, the API test also allow for much more complex scenarios when it comes to +building inferiors. Every tests has its own Makefile, most of them only a few +lines long. A shared Makefile (``Makefile.rules``) with about a thousand lines +of rules takes care of most if not all of the boiler plate, while individual +make files can be used to build more advanced tests. + +Another things this enables is having different variants for the same test +case. By default, we run every test for all 3 debug info formats, so once with +DWARF from the object files, once with gmodules and finally with a dSYM on +macOS or split DWARF (DWO) on Linux. But there are many more things we can test +that are orthogonal to the test itself. On GreenDragon we have a matrix bot +that runs the test suite under different configurations, with older host +compilers and different DWARF versions. + +As you can imagine, this quickly lead to combinatorial explosion in the number +of variants. It's very tempting to add more variants because it's an easy way +to increase test coverage. It doesn't scale. It's cheap in developer time, but +extremely expensive in terms of runtime and maintenance time because it +multiplies the overhead. + +The key take away is that the different variants don't prevent the need for +focused tests. So relying on it to test say DWARF5 is a really bad idea. +Instead you should write tests that check the specific DWARF5 feature, and have +the variant as a nice-to-have. + +In conclusion, you'll want to opt for an API test to test the API itself or +when you need the expressively, either for the test case itself or for the +program being debugged. The fact that the API tests work with different +variants mean that more general tests should be API tests, so that they can be +run against the different variants. -.. contents:: - :local: Running The Tests -----------------
_______________________________________________ lldb-commits mailing list lldb-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
