Re: [PATCH] Documentation: kunit: add tips for running KUnit
On Mon, Apr 12, 2021 at 1:42 PM Brendan Higgins wrote: > > On Mon, Apr 12, 2021 at 10:27 AM Daniel Latypov wrote: > > > > hOn Fri, Apr 9, 2021 at 9:10 PM David Gow wrote: > > > > > > Thanks for writing this: it's good to have these things documented at > > > last! > > > > > > There are definitely a few things this document points out which still > > > need deciding, which does make this document lean a bit into "design > > > discussion" territory in a few of the notes. This doesn't bother me -- > > > it's an accurate description of the state of things -- but I wouldn't > > > want this documentation held up too long because of these sorts of > > > TODOs (and can definitely see how having too many of them might > > > discourage KUnit use a bit). Particularly things like the > > > ".kunitconfig" fragment file feature stuff: I feel that's something > > > better discussed on patches adding/using the feature than in the > > > documentation / reviews of the documentation, so I'd rather drop or > > > simplify those '..note:'s than bokeshed about it here (something I'm a > > > little guilty of below). > > > > I don't think we'll actually make progress on any of those in the near > > future though. > > So I figured it'd be best to accurately represent the state of the > > world ~somewhere. > > > > But it did feel a bit strange to do it here, so I'm not against removing it. > > I actually like the accurate and upfront way that you spelled these things > out. > > > > Otherwise, a few minor comments and nitpicks: > > > > > > -- David > > > > > > On Sat, Apr 10, 2021 at 2:01 AM Daniel Latypov > > > wrote: > > > > > > > > This is long overdue. > > > > > > > > There are several things that aren't nailed down (in-tree > > > > .kunitconfig's), or partially broken (GCOV on UML), but having them > > > > documented, warts and all, is better than having nothing. > > > > > > > > This covers a bunch of the more recent features > > > > * kunit_filter_glob > > > > * kunit.py run --kunitconfig > > > > * kunit.py run --alltests > > > > * slightly more detail on building tests as modules > > > > * CONFIG_KUNIT_DEBUGFS > > > > > > > > By my count, the only headline features now not mentioned are the KASAN > > > > integration and KernelCI json output support (kunit.py run --json). > > > > > > > > And then it also discusses how to get code coverage reports under UML > > > > and non-UML since this is a question people have repeatedly asked. > > > > > > > > Non-UML coverage collection is no differnt from normal, but we should > > > > probably explicitly call thsi out. > > > > > > Nit: typos in 'different' and 'this'. > > Fixed. > > > > > > > > > > > As for UML, I was able to get it working again with two small hacks.* > > > > E.g. with CONFIG_KUNIT=y && CONFIG_KUNIT_ALL_TESTS=y > > > > Overall coverage rate: > > > > lines..: 15.1% (18294 of 120776 lines) > > > > functions..: 16.8% (1860 of 11050 functions) > > > > > > > > *Switching to use gcc/gcov-6 and not using uml_abort(). > > > > I've documented these hacks in "Notes" but left TODOs for > > > > brendanhigg...@google.com who tracked down the runtime issue in GCC. > > > > To be clear: these are not issues specific to KUnit, but rather to UML. > > > > > > (We should probably note where uml_abort() needs to be replaced if > > > we're mentioning this, though doing so below in the more detailed > > > section may be more useful.) > > > > Updated to > > *Using gcc/gcov-6 and not using uml_abort() in os_dump_core(). > > > > I figured we'd be more precise in the documentation itself. > > > > > > > > > > > > > Signed-off-by: Daniel Latypov > > > > --- > > > > Documentation/dev-tools/kunit/index.rst | 1 + > > > > .../dev-tools/kunit/running_tips.rst | 278 ++ > > > > Documentation/dev-tools/kunit/start.rst | 2 + > > > > 3 files changed, 281 insertions(+) > > > > create mode 100644 Documentation/dev-tools/kunit/running_tips.rst > > > > > > > > diff --git a/Documentation/dev-tools/kunit/index.rst > > > > b/Documentation/dev-tools/kunit/index.rst > > > > index 848478838347..7f7cf8d2ab20 100644 > > > > --- a/Documentation/dev-tools/kunit/index.rst > > > > +++ b/Documentation/dev-tools/kunit/index.rst > > > > @@ -14,6 +14,7 @@ KUnit - Unit Testing for the Linux Kernel > > > > style > > > > faq > > > > tips > > > > + running_tips > > > > > > > > What is KUnit? > > > > == > > > > diff --git a/Documentation/dev-tools/kunit/running_tips.rst > > > > b/Documentation/dev-tools/kunit/running_tips.rst > > > > new file mode 100644 > > > > index ..d38e665e530f > > > > --- /dev/null > > > > +++ b/Documentation/dev-tools/kunit/running_tips.rst > > > > @@ -0,0 +1,278 @@ > > > > +.. SPDX-License-Identifier: GPL-2.0 > > > > + > > > > + > > > > +Tips For Running KUnit Tests > > > > + > > > > + > > > > +Using ``kunit.py run`` ("kunit tool")
Re: [PATCH] Documentation: kunit: add tips for running KUnit
On Mon, Apr 12, 2021 at 10:27 AM Daniel Latypov wrote: > > hOn Fri, Apr 9, 2021 at 9:10 PM David Gow wrote: > > > > Thanks for writing this: it's good to have these things documented at last! > > > > There are definitely a few things this document points out which still > > need deciding, which does make this document lean a bit into "design > > discussion" territory in a few of the notes. This doesn't bother me -- > > it's an accurate description of the state of things -- but I wouldn't > > want this documentation held up too long because of these sorts of > > TODOs (and can definitely see how having too many of them might > > discourage KUnit use a bit). Particularly things like the > > ".kunitconfig" fragment file feature stuff: I feel that's something > > better discussed on patches adding/using the feature than in the > > documentation / reviews of the documentation, so I'd rather drop or > > simplify those '..note:'s than bokeshed about it here (something I'm a > > little guilty of below). > > I don't think we'll actually make progress on any of those in the near > future though. > So I figured it'd be best to accurately represent the state of the > world ~somewhere. > > But it did feel a bit strange to do it here, so I'm not against removing it. I actually like the accurate and upfront way that you spelled these things out. > > Otherwise, a few minor comments and nitpicks: > > > > -- David > > > > On Sat, Apr 10, 2021 at 2:01 AM Daniel Latypov wrote: > > > > > > This is long overdue. > > > > > > There are several things that aren't nailed down (in-tree > > > .kunitconfig's), or partially broken (GCOV on UML), but having them > > > documented, warts and all, is better than having nothing. > > > > > > This covers a bunch of the more recent features > > > * kunit_filter_glob > > > * kunit.py run --kunitconfig > > > * kunit.py run --alltests > > > * slightly more detail on building tests as modules > > > * CONFIG_KUNIT_DEBUGFS > > > > > > By my count, the only headline features now not mentioned are the KASAN > > > integration and KernelCI json output support (kunit.py run --json). > > > > > > And then it also discusses how to get code coverage reports under UML > > > and non-UML since this is a question people have repeatedly asked. > > > > > > Non-UML coverage collection is no differnt from normal, but we should > > > probably explicitly call thsi out. > > > > Nit: typos in 'different' and 'this'. > Fixed. > > > > > > > > As for UML, I was able to get it working again with two small hacks.* > > > E.g. with CONFIG_KUNIT=y && CONFIG_KUNIT_ALL_TESTS=y > > > Overall coverage rate: > > > lines..: 15.1% (18294 of 120776 lines) > > > functions..: 16.8% (1860 of 11050 functions) > > > > > > *Switching to use gcc/gcov-6 and not using uml_abort(). > > > I've documented these hacks in "Notes" but left TODOs for > > > brendanhigg...@google.com who tracked down the runtime issue in GCC. > > > To be clear: these are not issues specific to KUnit, but rather to UML. > > > > (We should probably note where uml_abort() needs to be replaced if > > we're mentioning this, though doing so below in the more detailed > > section may be more useful.) > > Updated to > *Using gcc/gcov-6 and not using uml_abort() in os_dump_core(). > > I figured we'd be more precise in the documentation itself. > > > > > > > > > Signed-off-by: Daniel Latypov > > > --- > > > Documentation/dev-tools/kunit/index.rst | 1 + > > > .../dev-tools/kunit/running_tips.rst | 278 ++ > > > Documentation/dev-tools/kunit/start.rst | 2 + > > > 3 files changed, 281 insertions(+) > > > create mode 100644 Documentation/dev-tools/kunit/running_tips.rst > > > > > > diff --git a/Documentation/dev-tools/kunit/index.rst > > > b/Documentation/dev-tools/kunit/index.rst > > > index 848478838347..7f7cf8d2ab20 100644 > > > --- a/Documentation/dev-tools/kunit/index.rst > > > +++ b/Documentation/dev-tools/kunit/index.rst > > > @@ -14,6 +14,7 @@ KUnit - Unit Testing for the Linux Kernel > > > style > > > faq > > > tips > > > + running_tips > > > > > > What is KUnit? > > > == > > > diff --git a/Documentation/dev-tools/kunit/running_tips.rst > > > b/Documentation/dev-tools/kunit/running_tips.rst > > > new file mode 100644 > > > index ..d38e665e530f > > > --- /dev/null > > > +++ b/Documentation/dev-tools/kunit/running_tips.rst > > > @@ -0,0 +1,278 @@ > > > +.. SPDX-License-Identifier: GPL-2.0 > > > + > > > + > > > +Tips For Running KUnit Tests > > > + > > > + > > > +Using ``kunit.py run`` ("kunit tool") > > > += > > > + > > > +Running from any directory > > > +-- > > > + > > > +It can be handy to create a bash function like: > > > + > > > +.. code-block:: bash > > > + > > > + function run_kunit() { > > > + ( cd "$(git rev-parse
Re: [PATCH] Documentation: kunit: add tips for running KUnit
hOn Fri, Apr 9, 2021 at 9:10 PM David Gow wrote: > > Thanks for writing this: it's good to have these things documented at last! > > There are definitely a few things this document points out which still > need deciding, which does make this document lean a bit into "design > discussion" territory in a few of the notes. This doesn't bother me -- > it's an accurate description of the state of things -- but I wouldn't > want this documentation held up too long because of these sorts of > TODOs (and can definitely see how having too many of them might > discourage KUnit use a bit). Particularly things like the > ".kunitconfig" fragment file feature stuff: I feel that's something > better discussed on patches adding/using the feature than in the > documentation / reviews of the documentation, so I'd rather drop or > simplify those '..note:'s than bokeshed about it here (something I'm a > little guilty of below). I don't think we'll actually make progress on any of those in the near future though. So I figured it'd be best to accurately represent the state of the world ~somewhere. But it did feel a bit strange to do it here, so I'm not against removing it. > > Otherwise, a few minor comments and nitpicks: > > -- David > > On Sat, Apr 10, 2021 at 2:01 AM Daniel Latypov wrote: > > > > This is long overdue. > > > > There are several things that aren't nailed down (in-tree > > .kunitconfig's), or partially broken (GCOV on UML), but having them > > documented, warts and all, is better than having nothing. > > > > This covers a bunch of the more recent features > > * kunit_filter_glob > > * kunit.py run --kunitconfig > > * kunit.py run --alltests > > * slightly more detail on building tests as modules > > * CONFIG_KUNIT_DEBUGFS > > > > By my count, the only headline features now not mentioned are the KASAN > > integration and KernelCI json output support (kunit.py run --json). > > > > And then it also discusses how to get code coverage reports under UML > > and non-UML since this is a question people have repeatedly asked. > > > > Non-UML coverage collection is no differnt from normal, but we should > > probably explicitly call thsi out. > > Nit: typos in 'different' and 'this'. Fixed. > > > > > As for UML, I was able to get it working again with two small hacks.* > > E.g. with CONFIG_KUNIT=y && CONFIG_KUNIT_ALL_TESTS=y > > Overall coverage rate: > > lines..: 15.1% (18294 of 120776 lines) > > functions..: 16.8% (1860 of 11050 functions) > > > > *Switching to use gcc/gcov-6 and not using uml_abort(). > > I've documented these hacks in "Notes" but left TODOs for > > brendanhigg...@google.com who tracked down the runtime issue in GCC. > > To be clear: these are not issues specific to KUnit, but rather to UML. > > (We should probably note where uml_abort() needs to be replaced if > we're mentioning this, though doing so below in the more detailed > section may be more useful.) Updated to *Using gcc/gcov-6 and not using uml_abort() in os_dump_core(). I figured we'd be more precise in the documentation itself. > > > > > Signed-off-by: Daniel Latypov > > --- > > Documentation/dev-tools/kunit/index.rst | 1 + > > .../dev-tools/kunit/running_tips.rst | 278 ++ > > Documentation/dev-tools/kunit/start.rst | 2 + > > 3 files changed, 281 insertions(+) > > create mode 100644 Documentation/dev-tools/kunit/running_tips.rst > > > > diff --git a/Documentation/dev-tools/kunit/index.rst > > b/Documentation/dev-tools/kunit/index.rst > > index 848478838347..7f7cf8d2ab20 100644 > > --- a/Documentation/dev-tools/kunit/index.rst > > +++ b/Documentation/dev-tools/kunit/index.rst > > @@ -14,6 +14,7 @@ KUnit - Unit Testing for the Linux Kernel > > style > > faq > > tips > > + running_tips > > > > What is KUnit? > > == > > diff --git a/Documentation/dev-tools/kunit/running_tips.rst > > b/Documentation/dev-tools/kunit/running_tips.rst > > new file mode 100644 > > index ..d38e665e530f > > --- /dev/null > > +++ b/Documentation/dev-tools/kunit/running_tips.rst > > @@ -0,0 +1,278 @@ > > +.. SPDX-License-Identifier: GPL-2.0 > > + > > + > > +Tips For Running KUnit Tests > > + > > + > > +Using ``kunit.py run`` ("kunit tool") > > += > > + > > +Running from any directory > > +-- > > + > > +It can be handy to create a bash function like: > > + > > +.. code-block:: bash > > + > > + function run_kunit() { > > + ( cd "$(git rev-parse --show-toplevel)" && > > ./tools/testing/kunit/kunit.py run $@ ) > > + } > > + > > +.. note:: > > + Early versions of ``kunit.py`` (before 5.6) didn't work unless run > > from > > + the kernel root, hence the use of a subshell and ``cd``. > > + > > +Running a subset of tests > > +- > > + > > +``kunit.py run`` accepts an optional glob
Re: [PATCH] Documentation: kunit: add tips for running KUnit
Thanks for writing this: it's good to have these things documented at last! There are definitely a few things this document points out which still need deciding, which does make this document lean a bit into "design discussion" territory in a few of the notes. This doesn't bother me -- it's an accurate description of the state of things -- but I wouldn't want this documentation held up too long because of these sorts of TODOs (and can definitely see how having too many of them might discourage KUnit use a bit). Particularly things like the ".kunitconfig" fragment file feature stuff: I feel that's something better discussed on patches adding/using the feature than in the documentation / reviews of the documentation, so I'd rather drop or simplify those '..note:'s than bokeshed about it here (something I'm a little guilty of below). Otherwise, a few minor comments and nitpicks: -- David On Sat, Apr 10, 2021 at 2:01 AM Daniel Latypov wrote: > > This is long overdue. > > There are several things that aren't nailed down (in-tree > .kunitconfig's), or partially broken (GCOV on UML), but having them > documented, warts and all, is better than having nothing. > > This covers a bunch of the more recent features > * kunit_filter_glob > * kunit.py run --kunitconfig > * kunit.py run --alltests > * slightly more detail on building tests as modules > * CONFIG_KUNIT_DEBUGFS > > By my count, the only headline features now not mentioned are the KASAN > integration and KernelCI json output support (kunit.py run --json). > > And then it also discusses how to get code coverage reports under UML > and non-UML since this is a question people have repeatedly asked. > > Non-UML coverage collection is no differnt from normal, but we should > probably explicitly call thsi out. Nit: typos in 'different' and 'this'. > > As for UML, I was able to get it working again with two small hacks.* > E.g. with CONFIG_KUNIT=y && CONFIG_KUNIT_ALL_TESTS=y > Overall coverage rate: > lines..: 15.1% (18294 of 120776 lines) > functions..: 16.8% (1860 of 11050 functions) > > *Switching to use gcc/gcov-6 and not using uml_abort(). > I've documented these hacks in "Notes" but left TODOs for > brendanhigg...@google.com who tracked down the runtime issue in GCC. > To be clear: these are not issues specific to KUnit, but rather to UML. (We should probably note where uml_abort() needs to be replaced if we're mentioning this, though doing so below in the more detailed section may be more useful.) > > Signed-off-by: Daniel Latypov > --- > Documentation/dev-tools/kunit/index.rst | 1 + > .../dev-tools/kunit/running_tips.rst | 278 ++ > Documentation/dev-tools/kunit/start.rst | 2 + > 3 files changed, 281 insertions(+) > create mode 100644 Documentation/dev-tools/kunit/running_tips.rst > > diff --git a/Documentation/dev-tools/kunit/index.rst > b/Documentation/dev-tools/kunit/index.rst > index 848478838347..7f7cf8d2ab20 100644 > --- a/Documentation/dev-tools/kunit/index.rst > +++ b/Documentation/dev-tools/kunit/index.rst > @@ -14,6 +14,7 @@ KUnit - Unit Testing for the Linux Kernel > style > faq > tips > + running_tips > > What is KUnit? > == > diff --git a/Documentation/dev-tools/kunit/running_tips.rst > b/Documentation/dev-tools/kunit/running_tips.rst > new file mode 100644 > index ..d38e665e530f > --- /dev/null > +++ b/Documentation/dev-tools/kunit/running_tips.rst > @@ -0,0 +1,278 @@ > +.. SPDX-License-Identifier: GPL-2.0 > + > + > +Tips For Running KUnit Tests > + > + > +Using ``kunit.py run`` ("kunit tool") > += > + > +Running from any directory > +-- > + > +It can be handy to create a bash function like: > + > +.. code-block:: bash > + > + function run_kunit() { > + ( cd "$(git rev-parse --show-toplevel)" && > ./tools/testing/kunit/kunit.py run $@ ) > + } > + > +.. note:: > + Early versions of ``kunit.py`` (before 5.6) didn't work unless run > from > + the kernel root, hence the use of a subshell and ``cd``. > + > +Running a subset of tests > +- > + > +``kunit.py run`` accepts an optional glob argument to filter tests. Currently > +this only matches against suite names, but this may change in the future. > + > +Say that we wanted to run the sysctl tests, we could do so via: > + > +.. code-block:: bash > + > + $ echo -e 'CONFIG_KUNIT=y\nCONFIG_KUNIT_ALL_TESTS=y' > > .kunit/.kunitconfig > + $ ./tools/testing/kunit/kunit.py run 'sysctl*' > + > +We're paying the cost of building more tests than we need this way, but it's > +easier than fiddling with ``.kunitconfig`` files or commenting out > +``kunit_suite``'s. > + > +However, if we wanted to define a set of tests in a less ad hoc way, the next > +tip is useful. > + > +Defining a set of tests >
[PATCH] Documentation: kunit: add tips for running KUnit
This is long overdue. There are several things that aren't nailed down (in-tree .kunitconfig's), or partially broken (GCOV on UML), but having them documented, warts and all, is better than having nothing. This covers a bunch of the more recent features * kunit_filter_glob * kunit.py run --kunitconfig * kunit.py run --alltests * slightly more detail on building tests as modules * CONFIG_KUNIT_DEBUGFS By my count, the only headline features now not mentioned are the KASAN integration and KernelCI json output support (kunit.py run --json). And then it also discusses how to get code coverage reports under UML and non-UML since this is a question people have repeatedly asked. Non-UML coverage collection is no differnt from normal, but we should probably explicitly call thsi out. As for UML, I was able to get it working again with two small hacks.* E.g. with CONFIG_KUNIT=y && CONFIG_KUNIT_ALL_TESTS=y Overall coverage rate: lines..: 15.1% (18294 of 120776 lines) functions..: 16.8% (1860 of 11050 functions) *Switching to use gcc/gcov-6 and not using uml_abort(). I've documented these hacks in "Notes" but left TODOs for brendanhigg...@google.com who tracked down the runtime issue in GCC. To be clear: these are not issues specific to KUnit, but rather to UML. Signed-off-by: Daniel Latypov --- Documentation/dev-tools/kunit/index.rst | 1 + .../dev-tools/kunit/running_tips.rst | 278 ++ Documentation/dev-tools/kunit/start.rst | 2 + 3 files changed, 281 insertions(+) create mode 100644 Documentation/dev-tools/kunit/running_tips.rst diff --git a/Documentation/dev-tools/kunit/index.rst b/Documentation/dev-tools/kunit/index.rst index 848478838347..7f7cf8d2ab20 100644 --- a/Documentation/dev-tools/kunit/index.rst +++ b/Documentation/dev-tools/kunit/index.rst @@ -14,6 +14,7 @@ KUnit - Unit Testing for the Linux Kernel style faq tips + running_tips What is KUnit? == diff --git a/Documentation/dev-tools/kunit/running_tips.rst b/Documentation/dev-tools/kunit/running_tips.rst new file mode 100644 index ..d38e665e530f --- /dev/null +++ b/Documentation/dev-tools/kunit/running_tips.rst @@ -0,0 +1,278 @@ +.. SPDX-License-Identifier: GPL-2.0 + + +Tips For Running KUnit Tests + + +Using ``kunit.py run`` ("kunit tool") += + +Running from any directory +-- + +It can be handy to create a bash function like: + +.. code-block:: bash + + function run_kunit() { + ( cd "$(git rev-parse --show-toplevel)" && ./tools/testing/kunit/kunit.py run $@ ) + } + +.. note:: + Early versions of ``kunit.py`` (before 5.6) didn't work unless run from + the kernel root, hence the use of a subshell and ``cd``. + +Running a subset of tests +- + +``kunit.py run`` accepts an optional glob argument to filter tests. Currently +this only matches against suite names, but this may change in the future. + +Say that we wanted to run the sysctl tests, we could do so via: + +.. code-block:: bash + + $ echo -e 'CONFIG_KUNIT=y\nCONFIG_KUNIT_ALL_TESTS=y' > .kunit/.kunitconfig + $ ./tools/testing/kunit/kunit.py run 'sysctl*' + +We're paying the cost of building more tests than we need this way, but it's +easier than fiddling with ``.kunitconfig`` files or commenting out +``kunit_suite``'s. + +However, if we wanted to define a set of tests in a less ad hoc way, the next +tip is useful. + +Defining a set of tests +--- + +``kunit.py run`` (along with ``build``, and ``config``) supports a +``--kunitconfig`` flag. So if you have a set of tests that you want to run on a +regular basis (especially if they have other dependencies), you can create a +specific ``.kunitconfig`` for them. + +E.g. kunit has own for its tests: + +.. code-block:: bash + + $ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit/.kunitconfig + +Alternatively, if you're following the convention of naming your +file ``.kunitconfig``, you can just pass in the dir, e.g. + +.. code-block:: bash + + $ ./tools/testing/kunit/kunit.py run --kunitconfig=lib/kunit + +.. note:: + This is a relatively new feature (5.12+) so we don't have any + conventions yet about on what files should be checked in versus just + kept around locally. But if the tests don't have any dependencies + (beyond ``CONFIG_KUNIT``), it's probably not worth writing and + maintaining a ``.kunitconfig`` fragment. Running with + ``CONFIG_KUNIT_ALL_TESTS=y`` is probably easier. + +.. note:: + Having ``.kunitconfig`` fragments in a parent and child directory is + iffy. There's discussion about adding an "import" statement in these + files to make it possible to have a top-level config run tests from all + child directories. But that would mean